ATD Links Archive
Issue Map
ATD Links Archive
ATD Links

ROI Performance Evaluation at AgCo

AgCo, a California agricultural organization that specializes in producing materials for the production of woodwind reeds for the saxophone and clarinet, has between five and 50 employees (seasonally dependent) and annual sales of less than $500,000. Production starts with the harvest of cane― a plant similar to bamboo―and continues with subsequent processing of the cane. After the cane is properly prepared, it can be cut into small tubes, which are split along their length into quarters called “splits.”  

Units of measurement for AgCo include the number of cane needed to produce a pound of tubes and the quantity of splits produced per pound of tubes (splits per pound). Splits production at AgCo was underperforming at a rate of 48.86 splits per pound, which was 12.8 percent lower than the goal of 56.05 splits per pound. That required 14 percent more tubes for production than normal. As a result, the overall splits margin was depressed by about 39 percent, which seriously affected company profits.  


A training and performance improvement consultant identified the root cause of the performance issue as inaccurate measurement of the thickness of the walls of the tubes prior to splits production. Inaccuracies with wall thickness measurements resulted in a proportion of material being erroneously discarded, since it appeared too thin. Workers demonstrated good split production techniques and knowledge, and the inaccuracies were ultimately traced back to poorly maintained thickness gauges that were not performing properly.  

The business owner typically calibrated and repaired gauges as required, but regularly relied on the workers to notify him of changes in gauge accuracy. Gauge accuracies underpinning the performance issue were overlooked by both the workers (who had no knowledge of gauge calibration) and the owner (who was experienced with gauge calibration). Thus, a gap in the gauge calibration protocol led to mechanical issues with the tools needed to perform. 

The problem was significant because the splits production unit kept key workers active and on the job at slow times between harvesting and processing. As such, the key workers remained with the AgCo year-round, reducing the risk that they would engage in other agricultural work and not return to the company. Keeping the key workers year-round reduced costs related to training replacements, and also significantly reduced costs of training for all other AgCo labor positions, since the key workers were completely cross-trained with all of the labor positions in the company.  

The key workers reduced the overall costs of initial training and retraining at the start of harvesting operations, when up to 45 untrained workers might join the company. Despite their importance, however, the profitability of the splits operations had to be sufficient to cover all associated operating costs.

Solutions and results

Two solutions were developed to improve splits production. First, a program for gauge calibration was established to support production efficiency. The splits operators were required to check their own gauges at the start and end of any work shift, at the start and end of any break, and whenever the worker felt it was needed. Second, a gauge calibration training initiative targeted the splits workers. Upon completion of the training, the workers were able to independently determine if the gauges they were using were accurate and performing according to expectations, and were thus able to support the gauge calibration program. The training was effective, and splits production performance improved from 48.87 splits per pound to 56.59 splits per pound.


To explore the effectiveness of the training initiatives, the ROI Method was applied. The ROI Method is a systematic and methodical measurement and evaluation approach that focuses on five levels of data and evaluation thereof to determine the effectiveness of a performance improvement initiative, according to Jack and Patti Phillips. The five levels of evaluation are reaction (participant feedback to the training), learning (learning, confidence, and understanding derived by the participants), application (how participants use what they learned from the initiative), business impact, (how the training affects the performance of the business), and ROI (the return-on-investment of the training, in terms of benefits relative to total costs). 

Objectives for each level of evaluation were identified, data collection tools and analytical approaches were developed, and an approach to isolate the effects of the program was devised. With data for the ROI evaluation in hand, a credible method to convert it into monetary values was developed, the costs of the program were totaled, and the actual return on investment of the program itself was calculated. With the ROI value of the program established, intangible benefits (non-monetary) were considered, issues related to the program were reviewed, and the evaluation results were prepared for appropriate communication to program stakeholders.

Influencing variables, trend line isolation, and data collection

The variables that could influence AgCo production data (or contribute to its improvement) included the accuracy of the gauges and regularity of gauge calibration (the focus of the initiative), gauge maintenance status (wear and tear), changes with the workers or quality standards, and changes in raw material quality. No changes had occurred with the overall maintenance status of the gauges, no workers had been changed, product quality standards remained the same, and raw materials exhibited no variability beyond minor expected fluctuations. In essence, only the initiative-related factors of gauge accuracy and the regularity of gauge calibration had changed between the pre- to post-solution periods. 

The purpose of the gauge calibration initiative was to improve splits production output by using more accurate measuring devices, thus increasing the number of splits produced per pound of starting material. Trend line isolation of the splits output data was accomplished by quantifying the number of splits produced for each pound of starting material used to allow comparisons between the pre- and post-initiative periods. The trend line compared pre-initiative projected splits production against post-initiative actual splits production.  

Obtaining the splits production data was highly feasible, since the workers reported this data every four hours in their normal production routine. A supervisor (or a material handler) verified splits output data, and so credibility and accuracy of the data was very high. Furthermore, the data was regularly collected as part of the work cycle, so obtaining it presented no disruption to the work environment, and its collection occurred at virtually no additional cost in terms of time and money.

Evaluation results

Final results indicated that the training initiative was successful at all levels of evaluation, and ultimately returned splits production to rates within expected norms, according to historical metrics. Evaluation results are summarized in Table 1 and indicate that splits production rates were improved to historical production trends, according to key business metrics.  



A trend line analysis described a significant difference in the projected continued production (no training initiative and the actual production (after the initiative) (see Figure 2).


Monetary value of benefits and costs

AgCo’s training initiatives were designed to return splits production rates to the standard rates derived from historical metrics. Splits per pound production rates were converted to a monetary value to determine splits profits. Profits, therefore, depended on the production costs of the splits and the sales value of the splits produced. Pre- and post-initiative splits production data was collected on a daily basis starting 40 days prior to the training, and continued through the 40-plus day post-initiative evaluation period. The splits production data that was converted to monetary values included historical, and pre- and post-initiative splits production labor costs and raw material costs.

Post-initiative benefits. Calculations of the net profit, or post-initiative benefit, considered profits during the post-training period (≈$64,563) less predicted profits during the 40-day post-initiative period (≈$38,204, representing what would have been realized without the training improvements). The net profit, or benefit, directly derived from the initiative was ≈$26,359.

Program costs. Only specific concrete costs were attributed to the evaluation, including all direct expenses related to needs analyses, program design, development, execution, and ROI evaluation. In addition, the business profits decreased during the initiative, and such losses were to be considered a cost of the program. Documented costs involved with the training totaled $9,159.


ROI calculation. Using the established post-initiative monetary benefits and program costs, both program ROI and annualized ROI were calculated.  

ROI = Net Program Benefits ÷ Program Costs ´ 100, where the Net Program Benefits = (Net Post-Initiative Profits – Program Costs) (according to a 2007 Phillips book titled, “The Value of Learning”). Thus, ROI = (Net Post-Training Profits – Program Costs) ÷ Program Costs ´ 100.  

The Program ROI calculates the ROI of the initiative upon conclusion of the 40-day post-initiative evaluation period. An annualized ROI forecast calculates the ROI anticipated over the course of a full year, which normally would include two 40-day production periods. Table 2 summarizes the ROI calculations.



The program ROI was 188 percent, indicating that each dollar spent on the program (program costs) was recovered, and an additional $1.88 was captured as a profit to the business. Clearly, AgCo’s training program was effective in improving the splits per pound production rate. The projected annual ROI was 475 percent, which indicated that in addition to recovering the program costs the fully loaded program cost would be returned as profit to the business 4.75 times. The projected annual ROI further validates the selected interventions and associated process improvement efforts.

Intangible benefits. Intangible benefits―measures purposely not converted to money―were identified in discussions with the business owner, site supervisor, and workers. Since credible value is not easily assigned to intangibles, these items could not to be included in calculations for ROI purposes. Identified intangibles were teamwork, commitment, communication, and job satisfaction.  

A final identified intangible that could probably have been measured, but would be difficult to convert, was job knowledge not directly applicable to splits operations. Since splits workers are retained year round and are cross-trained in all aspects of AgCo operations, they carry knowledge that is used for faster training of new hires for harvesting and processing cycles, potentially reducing the start-up costs related to those cycles. 

Without the intangibles associated with the splits operations, the splits operations were simply profitable. With intangibles considered, in conjunction with splits production profits, the intrinsic value of the workers is underscored, and their long-term impact and importance to the organization can be better appreciated.

Communication of results

From the start of the initiative to end of the evaluation, the workers and the business owner prized good communication. Personal involvement by the participants improved their opinions toward the program, which resulted in great anticipation for the final results. The close involvement of all in the initiative allowed a common dialogue to develop, so that final results were easily communicated and understood by all.  

The full summary of evaluation results was presented after the end of the most recent processing activities. Since the information was sensitive, only the business owner, site supervisor, and splits operators were included in the communication efforts (client stakeholders were excluded). The results communication was delivered by the training and performance improvement consultant, who was responsible for organizing and implementing the initiative and evaluation.  

The results were communicated in an informal meeting on the splits production site, and required the most basic of media for presentation and display (printed flip charts, posters, and handouts). Given that the audience was familiar with the information and did not require a complicated delivery system, “simple” was not only better, but was preferred. A general overview and singe-page, written summary sufficed.

Overcoming barriers

Early in the training process, implementation timing, commitment, leadership, communication, and resources were considered and addressed to ensure the AgCo performance improvement project moved ahead smoothly. Timing and worker commitment stood out as key areas for concern, since the period for producing splits is very short and workers in the operation may feel that their efforts will not be fully recognized until too late. Additionally, since poor communication seemed to undermine previous improvement efforts, communication with the stakeholders was a high priority. 

To minimize problems associated with time and commitment, AgCo was very proactive with leadership, communication, and resources issues, and directly addressed all concerns for time and commitment. A learning professional was selected to be the leader of the initiative, because it she was widely viewed as an unbiased advocate of accountability and progress. The stakeholders had prior, positive experience with the training and performance improvement consultant, and were uniformly in favor of his participation and guidance. She opened communication to allow very clear and direct dialogue between all of the involved AgCo parties, thus improving trust and understanding between all. Resources were prepared to ensure material success with the program (new tools and extra gauges were made specifically for the project). Once it became clear that there would be appropriate time to implement the program, and that the findings were likely to dramatically improve future work efforts, workers had good incentive to fully support the program.

Continued improvement

AgCo’s gauge calibration training to improve splits production was a major success for the company. Not only was splits production returned to standards, but a path was developed to retain and stabilize the knowledge base in the organization. The initiative required a change in the way that improvements were made at AgCo (better communication and a dedicated lead for the project), which resulted in numerous intangible benefits, including greater employee loyalty and commitment. AgCo remains dedicated to ensuring that the splits workers operate with the spirit of collaboration, thus avoiding the potential that they may become protective of their knowledge against the health of the organization.

Performance improvement

AgCo is an agricultural operation that, like other agricultural operations, must balance investments in worker development against constant turnover. Performance improvement at AgCo was typically accomplished using a business standards enforcement approach. Workers failing to meet the minimum standards were generally released and replaced, leading to a continuous cycle of hiring, training, and measuring that mirrored the constant operational cycle of harvesting, splitting, and processing.  

In essence, hiring and training a replacement for a poorly-performing worker is the same thing as simply resolving performance issues with training, regardless of the root cause of the issue itself. AgCo’s current approach to performance improvement includes greater reliance on worker contributions, feedback, and suggestions for environmental improvements, in addition to investment into worker knowledge and training. Rather than address perceived training needs of the problem, AgCo now works to address underlying performance needs of a problem. 

As a result, AgCo now relies more on workers to identify issues that contribute to performance problems, and rewards those workers contributing positively toward improvement. Within the operational structure, the workers are now the performer stakeholders driving toward the specific process-level goals that are pertinent at any given time. A worker’s ability to contribute varies, depending on the particular process goals (harvesting, processing, or splitting, depending on the season).  

Since splitting operations bridge the harvesting and processing seasons, they are critical to helping maintain and stabilize the employee base and associated knowledge base. Performance improvement at AgCo, therefore, now starts with the splitting operations, where the key performers are the splits producers. These workers offer not only a good return on the investment in terms of products produced, but also in terms of the capacity to maintain the operational business knowledge, which, at the least, further reduces costs of training (in terms of time and money) for harvesting and processing activities.

AgCo performance and learning strategies

AgCo’s workplace performance has improved over the course of the gauge calibration initiative and evaluation. The success of this initiative generated increased dedication by all AgCo workers. The example set by the splits workers was a strong indicator for temporary workers. Leveraging the knowledge of the workers to help develop the training initiative not only helped isolate the splits production issue, but also led to excellent participant buy-in and, ultimately, cohesiveness. 

AgCo employed relatively simple learning strategies, including on-the-job skill development and shadowing (observing) for general knowledge transfer. However, more complicated tools for learning are now being implemented, including job aids and data-based performance monitoring for additional guidance to the workers, so that learning and improvements can be more focused. Furthermore, brainstorming sessions and meetings between the owner and the employees have been learning tools for all involved, leading to new ideas for operational improvements.

 Finally, with retained workers, the training programs for new employees have been dramatically improved, even though they have not yet been optimized. In general, AgCo has come a long way with the learning strategies initially implemented, and continued improvements are likely, provided that the company continues to practice what has been learned and utilizes the collaborative improvement tools that have been developed.

Lessons learned

The stakeholders agreed communication was a key to the success of the evaluation. Since the communication plan was clearly described, it helped to ensure effective distribution of appropriate information and results to the stakeholders. The training and performance improvement consultant helped to improve communication, as well, which lead to a successful initiative and evaluation. The enhanced communications increased the level of stakeholder commitment. 

The project was initiated with an ROI evaluation as a planned component. As a result, the entire evaluation effort was fairly easy and yielded highly applicable data at each evaluation level. If the project and the evaluation were not so tightly integrated, the evaluation data might have gone uncollected or unused. 

Finally, the impact-level objective was specific and tied directly to the performance gap. This alignment made the ROI evaluation more easily understood by the stakeholders, and that much more relevant. Since the stakeholders were also responsible for collecting the data, as they began to understand it better, they became better able to discern the impact of any minor fluctuations in performance on the overall program ROI. Greater understanding led to improvements in an array of intangibles that trickled through the AgCo structure―resulting in company changes that have benefited all involved.

About the Author
Michael Nicholson is a certified performance technologist and a certified ROI professional, and he earned an MS in education in training and performance improvement from Capella University. He teaches college biology and microbiology, and is a performance improvement consultant, addressing domestic and international issues in agriculture and biotechnology.
Be the first to comment
Sign In to Post a Comment
Sorry! Something went wrong on our end. Please try again later.