Last month we discussed how the typical training assessments we give after training, such as questions that tell us whether people can recall the content, aren’t much of an assessment at all. Do we really care if people can remember the name of the document? Or do we want to know something else?
Well, that’s the thing. First we have to figure out what we want to know. Take a look at the following questions. Which of these do you or your L&D organization need the answers to? (What will you do with the answers? If the answer is nothing, you possibly don’t need the answer. Collecting data and analyzing it take time.)
- Did people achieve the intended learning outcomes?
- Does the instruction have defects, unclear points, or other problems?
- How do people attain their own learning needs?
- How can we support individuals, departments, and the organization's learning needs?
- How do people use what they learned from a learning program?
- In what does ways instruction impact the organization?
These are just a few of the questions that L&D practitioners might want to answer under the domain of assessment and evaluation. So how do we answer them? The answers could be a book, but let’s talk about a few of the basics.
ASSESSMENT is the process of collecting data about individual knowledge and skills in order to make inferences (conclusions). (See Question 1 above.)
Testing is one method of assessment but recall questions aren’t great because they provide useless information. (I talked about this last month and provided some alternatives.) We can use data from tests (item analysis) to help us determine what’s wrong with the instruction or the test or we can ask people to tell us what isn’t clear or isn’t working. (See Question 2 above.)
L&D practitioners often use flawed assessments and rarely check to see whether instruction is problematic. Look around and you’ll see poorly written assessment items and casually set cut-off scores. Depending on the context, this could be a legal nightmare waiting to happen.
Assessments take time to write and take. We need to ask ourselves whether we need to know if people can pass a poorly written test with a casually set cut-off score. If you have good item writers and can justify your cut-off score and you want to know the answer to Question 1 via test questions, that’s an acceptable approach. (Sometimes we just need to know that people read something, so a checkbox will do.)
Rather than Question 1, perhaps, we are better off spending our limited time with Questions 5 and 6 from the list above. How do people use what they learned and how the learning impacted the organization. But that’s more about evaluation than assessment?
EVALUATION is analyzing assessment and other data to draw conclusions about the value of what we do.
One of the most popular evaluation models in training is Kirkpatrick’s model of four levels of evaluation: reaction, learning, on-the-job performance, and business results. The Kirkpatrick model has been the primary model for training evaluation for many years because it provides a systematic way to look at training outcomes. It has made extraordinarily valuable contributions to the field.
Dr. Reid Bates, professor of human resource and leadership development at Louisiana State University, wrote that any good evaluation framework must help us answer these questions:
- Was the program effective?
- What can we do to improve it?
Bates says the flaws in the Kirkpatrick evaluation model create problems towards answering those questions, especially when taken together. Flaws include creating perceptions that reaction measures can stand in for training outcomes (they can’t), creating an impression that the levels are causally linked (research shows they aren’t), and creating the appearance that each level provides more important data than the previous level (not necessarily).
Business and financial people often scoff at ROI (return on investment) because we simply cannot account for all of the variables that impact training. Bates discusses this in his paper, which is yet another problem with Level 4 measures.
The model provides some good information (especially when used as a classification system), but the flaws point out problems such as trainers expending resources to produce positive reactions rather than much more important outcomes such as evaluating and fixing instruction. And he points out that we must do these things.
So, how do we attain the elusive evaluating and fixing instruction (Questions 5 and 6)? I discussed some of this in last month’s article. Good advice comes from Robert Brinkerhoff’s latest book. And the good news is that the processes he discusses are not difficult, and puts us in touch with key stakeholders of our work.
What does he suggest? His model tells us how to identify the most and least successful trainees and conduct interviews with selected samples in order to gain critical information about the specific nature and scope of impact from training and the improvements that are needed. There is more to this process, of course, but the good news is that it is tangible, feasible, and impactful to leaders.
We find time for this by letting go of tasks that don’t need doing. (And here’s where I take off my Science of Learning hat and put on my long-time practitioner hat.) Stop building courses when a job aid or PDF will absolutely do. When no one cares about assessment information, but you need to know if someone has downloaded or read it, use a checkbox. Measure when it’s truly important.
I could spend a long time on assessment and evaluation (and would gladly because it’s one of my favorite topics) but I think we’ll move on next month to some of the basic elements research shows improves learning.
Tell me where I lost you or where you want to know more and I’ll recommend additional resources. And as usual, I’d love your thoughts!
Bates, R. (2004). "A critical analysis of evaluation practice: The Kirkpatrick model and the principle of beneficence." Evaluation and Program Planning, 27, 341–347.
Brinkerhoff, R. O. (2006). Telling training's story: Evaluation made simple, credible, and effective. Berrett-Koehler Publishers.
Kirkpatrick, D. L. (1976). Evaluation of training. In R. L. Craig (Ed.), Training and development handbook: A guide to human resource development. New York: McGraw Hill.
Shank, P. (February 06, 2015.) "Science of Learning 101: Can We Really Assess Learning?" ATD Science of Learning Blog.