You’ve got two courses with different approaches: design A and design B. Which one is better? Which design should you use as a model for future work? Which design can you point to as the better example of your team’s success? The answers to these questions can be found with metrics.
Start With Clear Learning Objectives
Pinpointing what you mean by training success can be the hardest part. What problems are you trying to solve? What does success look like? Herein lies the key to finding good metrics: you need to start from the end point—the goal.
The goal can come from a formal, quantitative needs assessment and gap analysis. However, in the real world, this often isn’t an option. A rigorous needs analysis can require the same time and investment as simply making the course and seeing “if it flies.”
More importantly, management may not support the cautious route of a formal needs analysis. Instead, you may have to work from an informal understanding of what the organization needs, and then iterate the solution. When you do that right, you have an agile planning process (the topic of a future post).
From the perspective of measuring outcomes, we won’t worry about how you derive your learning objectives. We only care that the objectives are clear, specific, and measurable—and reasonably related to needs.
Here are some examples of good learning objectives:
- Engineers will write specifications that meet the peer review standards defined in document XYZ.
- Sales staff will introduce Widget X into conversations with customers about scenario Y.
- Employees will accurately describe the company’s policy for data privacy.
Here are some not-so-good examples of learning objectives:
- Engineers will write good specifications. (This isn’t narrow enough to be measureable).
- Sales staff will be knowledgeable about Widget X. (This is measurable, but it isn’t the real goal).
- Reduce costs. (How many things contribute to cost reduction? 1000? And which of those are beyond the scope of your training? Yes, you should lay out a plausible connection between your learning objectives and a broad goal such as this one. But understand that you won’t be able to prove that your training caused a reduction in cost.)
- Improve quality. (Ditto everything above.)
Base Metrics on the Learning Objectives
With your learning objectives in hand, the rest of your work is mapped out for you. The type of metric will depend on the particular learning objective.
-
For specification writing, evaluate the quality of real specifications before and after training. If there’s already a peer review process in place, simply track the results of the peer reviews.
-
For sales-related objectives, you can track sales staff behavior through a questionnaire, before and after training. Better yet, track it through a CRM application that they’re already using.
- For the privacy policy objective, quiz employees on privacy policy as part of their online course. These quizzes can be embedded in the course as self-tests and tracked through SCORM. To assess knowledge transfer beyond short-term memory, survey the course takers at some point after the course, or give them a refresher course that includes a pre-test.
Get Ready for the Long Haul
Good metrics add a minimum of overhead. Ideally, they don’t add any overhead at all: existing peer reviews, an existing CRM logging tool, SCORM that’s already part of your course delivery system, and so forth. The less obtrusive and cheaper your metric is, the more likely it will be around for the long haul. That’s a good thing, because baseline data is a goldmine when you decide to try something new—like replacing design A with design B.
