ATD Blog
Track Success with Metrics
Fri Jan 23 2015
Content
You’ve got two courses with different approaches: design A and design B. Which one is better? Which design should you use as a model for future work? Which design can you point to as the better example of your team’s success? The answers to these questions can be found with metrics.
You’ve got two courses with different approaches: design A and design B. Which one is better? Which design should you use as a model for future work? Which design can you point to as the better example of your team’s success? The answers to these questions can be found with metrics.
Content
Start With Clear Learning Objectives
Start With Clear Learning Objectives
Content
Pinpointing what you mean by training success can be the hardest part. What problems are you trying to solve? What does success look like? Herein lies the key to finding good metrics: you need to start from the end point—the goal.
Pinpointing what you mean by training success can be the hardest part. What problems are you trying to solve? What does success look like? Herein lies the key to finding good metrics: you need to start from the end point—the goal.
Content
The goal can come from a formal, quantitative needs assessment and gap analysis. However, in the real world, this often isn’t an option. A rigorous needs analysis can require the same time and investment as simply making the course and seeing “if it flies.”
The goal can come from a formal, quantitative needs assessment and gap analysis. However, in the real world, this often isn’t an option. A rigorous needs analysis can require the same time and investment as simply making the course and seeing “if it flies.”
Content
More importantly, management may not support the cautious route of a formal needs analysis. Instead, you may have to work from an informal understanding of what the organization needs, and then iterate the solution. When you do that right, you have an agile planning process (the topic of a future post).
More importantly, management may not support the cautious route of a formal needs analysis. Instead, you may have to work from an informal understanding of what the organization needs, and then iterate the solution. When you do that right, you have an agile planning process (the topic of a future post).
Content
From the perspective of measuring outcomes, we won’t worry about how you derive your learning objectives. We only care that the objectives are clear, specific, and measurable—and reasonably related to needs.
From the perspective of measuring outcomes, we won’t worry about how you derive your learning objectives. We only care that the objectives are clear, specific, and measurable—and reasonably related to needs.
Content
Here are some examples of good learning objectives:
Here are some examples of good learning objectives:
Content
Engineers will write specifications that meet the peer review standards defined in document XYZ.
Engineers will write specifications that meet the peer review standards defined in document XYZ.
Content
Sales staff will introduce Widget X into conversations with customers about scenario Y.
Sales staff will introduce Widget X into conversations with customers about scenario Y.
Content
Employees will accurately describe the company’s policy for data privacy.
Employees will accurately describe the company’s policy for data privacy.
Content
Here are some not-so-good examples of learning objectives:
Here are some not-so-good examples of learning objectives:
Content
Engineers will write good specifications. (This isn’t narrow enough to be measureable).
Engineers will write good specifications. (This isn’t narrow enough to be measureable).
Content
Sales staff will be knowledgeable about Widget X. (This is measurable, but it isn’t the real goal).
Sales staff will be knowledgeable about Widget X. (This is measurable, but it isn’t the real goal).
Content
Reduce costs. (How many things contribute to cost reduction? 1000? And which of those are beyond the scope of your training? Yes, you should lay out a plausible connection between your learning objectives and a broad goal such as this one. But understand that you won’t be able to prove that your training caused a reduction in cost.)
Reduce costs. (How many things contribute to cost reduction? 1000? And which of those are beyond the scope of your training? Yes, you should lay out a plausible connection between your learning objectives and a broad goal such as this one. But understand that you won’t be able to prove that your training caused a reduction in cost.)
Content
Improve quality. (Ditto everything above.)
Improve quality. (Ditto everything above.)
Content
Base Metrics on the Learning Objectives
Base Metrics on the Learning Objectives
Content
With your learning objectives in hand, the rest of your work is mapped out for you. The type of metric will depend on the particular learning objective.
With your learning objectives in hand, the rest of your work is mapped out for you. The type of metric will depend on the particular learning objective.
Content
For specification writing, evaluate the quality of real specifications before and after training. If there’s already a peer review process in place, simply track the results of the peer reviews.
For specification writing, evaluate the quality of real specifications before and after training. If there’s already a peer review process in place, simply track the results of the peer reviews.
Content
For sales-related objectives, you can track sales staff behavior through a questionnaire, before and after training. Better yet, track it through a CRM application that they’re already using.
For sales-related objectives, you can track sales staff behavior through a questionnaire, before and after training. Better yet, track it through a CRM application that they’re already using.
Content
For the privacy policy objective, quiz employees on privacy policy as part of their online course. These quizzes can be embedded in the course as self-tests and tracked through SCORM. To assess knowledge transfer beyond short-term memory, survey the course takers at some point after the course, or give them a refresher course that includes a pre-test.
For the privacy policy objective, quiz employees on privacy policy as part of their online course. These quizzes can be embedded in the course as self-tests and tracked through SCORM. To assess knowledge transfer beyond short-term memory, survey the course takers at some point after the course, or give them a refresher course that includes a pre-test.
Content
Get Ready for the Long Haul
Get Ready for the Long Haul
Content
Good metrics add a minimum of overhead. Ideally, they don’t add any overhead at all: existing peer reviews, an existing CRM logging tool, SCORM that’s already part of your course delivery system, and so forth. The less obtrusive and cheaper your metric is, the more likely it will be around for the long haul. That’s a good thing, because baseline data is a goldmine when you decide to try something new—like replacing design A with design B.
Good metrics add a minimum of overhead. Ideally, they don’t add any overhead at all: existing peer reviews, an existing CRM logging tool, SCORM that’s already part of your course delivery system, and so forth. The less obtrusive and cheaper your metric is, the more likely it will be around for the long haul. That’s a good thing, because baseline data is a goldmine when you decide to try something new—like replacing design A with design B.