Reconciling economic, statistical, and system structure significance in field trial results

Locke A. Karriker, DVM, MS

Veterinary Diagnostic and Production Animal Medicine Department, College of Veterinary Medicine, Iowa State University; Tel: 515-294-2283; karriker@iastate.edu.

Cite as: Karriker LA. Reconciling economic, statistical, and system structure significance in field trial results. J Swine Health Prod. 2004;12(6):334-335.

Performance, carcass, and treatment frequency data were obtained in a field trial comparing two potential interventions (Treatments 1 and 2) and the current intervention for the herd (Control). This figure shows the relative profit per head ($US) as a result of all changes in the trial, calculated using the summary data and the system-specific, incremental values per head of changes in the parameters measured. Can the differences in value shown in the figure be extrapolated to the rest of the system or to all pigs in the production system?

Use of field trials that compare potential interventions or treatments in a swine production system can be a powerful tool both for management and validation of clinical research under pertinent conditions. Conducting these trials reliably is an investment in strict quality control, follow-up, and organization. Interpretation of the data set may be difficult in an environment where investment is required to implement the change, and an understanding of the potential return in financial terms is required.

The initial approach is often to total the economic differences represented in the trial. In this example, it would entail multiplying the incremental difference in the parameter averages by the value of the change in the system and totaling those for each treatment. For example, Treatment 1 resulted in a benefit of 0.2 lb decrease in feed:gain (F:G) compared to controls (Table 1). In this system, a 0.1 lb increase in F:G costs the producer $1.50 per head (Table 2; all currency in $US). When this difference is multiplied by the cost of $1.50 per 0.1 lb increase in F:G, the total change due to Treatment 1 is a benefit of $3.00 per head. Treatment 2 results in a 0.3-lb improvement in F:G, valued at $4.50 per head. Conversely, Treatment 1 was associated with greater mortality, which, when calculated similarly, would result in a cost of $0.65 per head. Totaling these costs and benefits for each treatment, including the cost of treatment, results in the totals in Figure 1. Treatment 2 appears to be the best treatment regime, with an economic advantage of $6.23 per head over the current treatment and $6.43 per head over Treatment 1. Most systems would implement a process that increased the value per head by this amount. Can this difference in value be extrapolated to the rest of the system or to all pigs in the production system? To answer that, statistical significance must be included in the analysis. Statistical analysis was performed using a one-way ANOVA test (JMP 5.1.1 software; SAS Institute, Cary, North Carolina).

Note that not all of the differences observed in the trial were statistically significant (Table 1). This may be the result of small sample size, high variability among replicates, or a true lack of impact on the variable by the treatment. Despite the potential financial significance of the differences, lack of statistical significance means that from a practical standpoint, we have little confidence that the same results will occur when these treatments are repeated or applied to the whole system. A more realistic or confident answer can be obtained by including only the statistically significant variables in the analysis. Figure 2 includes only the significant differences. Note that the potential difference in the values of the treatments disappears, and both provide value to the system relative to the current treatment.

Reconciling these differences in potential value may be troubling. The low-risk, conservative approach would be to make immediate decisions on implementation on the basis of the value of the parameters that were statistically different. Concurrently, perform a follow-up trial designed with greater statistical power to determine whether the larger economic difference in treatments is repeatable.

The temptation in field trials, especially when the potential mechanism of the intervention is poorly understood, is to collect all available outcome data and examine for differences. However, cost benefit economic evaluations should be confined to parameters whose benefit or cost will be realized by the production unit implementing the change. In this example, if the pigs changed ownership at processing, the change in number of loins qualifying for foreign markets would not impact the producer’s bottom line. Evaluation of only the parameters important to the producer yields Figure 3. Note that Treatment 2 remains the preferable intervention. However, the impact of statistical significance on the decision-making process is still relevant. When the economic impact of the parameters that are not statistically different is eliminated, the preferred intervention for the producer becomes Treatment 1 (Figure 4).

Field trials in production systems may be enormously valuable for comparing the cost benefit of interventions in a pertinent environment. Interpretation of data and calculation of potential cost and benefit must be conducted with respect to system structure and must integrate statistical and economic significance. Application of these base considerations opens the door for application of more advanced tools for data analysis and determining economic value. Assisting clients with realistic evaluations of interventions is another tool for veterinarians to use with clients that brings value and improves producer understanding of their production system.