Learning professionals are generally fluent, and in some cases expert, in the instructional design principles of analysis, design, develop, implement, and evaluate (ADDIE). Busy practitioners tend to excel in the middle (DDI) of the process, namely designing, developing, and implementing high-quality training solutions. Where we tend to be less proficient is at the front and back ends of the process, namely, analyzing first whether a training solution is the answer to the business problem at hand, determining what a successful learning intervention will look like, and creating an evaluation plan that shows whether we have hit the target. I know many professionals who get stressed at end-of-year or budget time because they so often just don’t have enough data to justify additional or even current spend. This post will get you started on a path toward a winning training metrics strategy.
Difficulties With Measuring Training
There are many reasons why measurement of training effectiveness is difficult. This is particularly acute in complex, regulated industries like life sciences (such as pharmaceuticals) and tech or engineering. Some of the challenges include:- Long business cycle times. In some industries like pharma, medical devices, and defense, product development times from idea to market can be years or decades. This creates data collection and reporting issues because the event is too old when relevant data becomes available.
- Confounding factors. When training is aimed at a business problem that is easy to isolate (such as manufacturing error rates or call center resolution times), it is easier to collect data on the effect of training. When this is not the case, various factors can make it very difficult to prove the training intervention had an effect. An example of this in the conduct of clinical trials, where you have multiple corporate entities working on the same study and the source of business problems is difficult to isolate.
- Lack of training professionals’ expertise in training metrics. Many practitioners are fluent in instructional design the same way we are fluent in our first language—we just “picked it up” over time. Fewer of us have formal ID education in our backgrounds, particularly in the most difficult yet most important area for training leaders to be expert in: evaluation.
- Lack of training professionals’ expertise in consulting skills. This front-end skill is critical to your success. “Taking an order” for a training program that will never fix the business problem is easy in the short-term, but not effective in the long-term. Fitting solutions to problems and gaining agreement up front about what looks good is critical.
A Winning Approach
Here are some things you can add to your development plan to become more fluent, and over time, expert in the “A” and “E” of the process. These are longer-term plans, but if carefully added to your professional development, can be done very successfully.- Continue to collect your typical dashboard metrics. Continue to gather Level 0 (usage, cost, other logistics), Level 1 (reaction/planned action), Level 2 (learning/assessment), and Levels 3-5 as you can. Dashboards are usually valuable to the business. Just make sure that the data collection and reporting efforts are worth the benefit you get from the exercise—don’t go “dash-overboard.”
- Develop your consulting skills. Whether you are a trainer, or particularly if you are a training leader, you will be much more successful by being an expert business partner with excellent consulting skills, able to design interventions that are fit for purpose and agree with your stakeholders on the evaluation strategy that is also fit for purpose. No metrics model can help you here; what looks good is what looks good to your stakeholder.
- Develop your evaluation skills. Become fluent in metrics models like Kirkpatrick and Phillips, but become expert in the development of metrics strategies.
These plans will take time for you to get the outside help you need and to actually learn and practice your new expert skills . . . but it will be well worth it. Remember, the definition of success of any training intervention is what you and your business stakeholder have agreed on.
