Assess the Proper Outcomes

Editor's Note: Ken Phillips passed away in mid-December. He was beloved by the L&D community. Read this blog post on his legacy.

Research shows that four out of five organizations use knowledge tests to evaluate their learning programs. However, most Level 2 knowledge tests consist mainly or entirely of recall test questions, which assess the recall of facts and information from the training program. In L&D, though, knowing is not enough. Training participants also must understand how to apply what they've learned. That's why tests that assess job application are a better option.

What it is

Job application test questions serve as a credible measure of participant learning post-training. They also serve as a credible measure of whether participants know how to apply the new information on the job.

How it works

The test score data can predict likely Level 3 job application results. For example, high test scores forecast that training participants will likely apply what they learned in training on the job because it shows that they learned and know how to apply the training material. In contrast, low test scores forecast that Level 3 job applications will likely be low because participants didn't learn the training material and didn't know how to apply it.

Further, the test score data can pinpoint the training program's effect on training transfer. For instance, suppose that the test score data is high, and the Level 3 evaluation data indicates that participants didn't apply what they learned after returning to their jobs. In that case, the test results demonstrate that the participants grasped the training content and knew how to apply it, but something hindered them from using their knowledge post-training. Possible factors include a time gap between training attendance and practical application or insufficient support from managers or colleagues regarding the training content.

Guidelines

To create job application test questions, start by selecting a learning objective from your target training program and consider what learners need to know and do to meet the objective. Then, identify real-life situations that reflect the application of the program content the objective c0vered.

If you are unfamiliar with the jobs the learners perform, a best practice is to interview several subject matter experts familiar with the learners' jobs and ask them to describe job-related situations. Choose one of the situations and write the stem (the question or problem) in one to three sentences. Identify plausible distractors; think of things someone who thought they knew the content associated with the learning objective may say or do if they didn't understand.

After creating the job application questions, your next task is to validate their quality.

Begin by administering the test to a sample of 15 to 20 participants. They could be the first 15 to 20 participants to attend the training program. Alternatively, if a large number of people have registered to attend the program, you could randomly choose individuals to take the test pre-training. The key is that those you select must be representative of the majority of the people attending the training program in terms of factors such as years of experience, tenure in current position, age, and gender.

The data you collect will enable you to assess how well each test item measures learner ability and how each test item distractor performed.

Difficulty index. This index measures the percentage of participants who answered a specific test item correctly. The percentage indicates the difficulty of the test item. Good test questions generally have a difficulty score between 30 percent and 70 percent, ranging from 0 percent (no one answered the item correctly) to 100 percent (everyone answered it correctly).

Percentages above 70 indicate that the test item may be too easy, and those below 30 suggest that the question may be too difficult. Any test items that fall outside the 30/70 range are candidates for rewriting. Follow the process in the checklist below to calculate the difficulty index score for each test question.

Distractor pattern analysis. This process indicates the presence of a distractor issue. If you find that participants consistently eliminate one or more response options as incorrect, then your response options may be too unrealistic or too easy. If that happens, you have changed the probability of someone guessing the correct answer. The indicators of a distractor issue are:

• A significant number of participants select the same incorrect response option. That suggests that the response option is too close to the correct answer and is a candidate for rewriting.

• A test item has one or more response options with zero or few selections. That shows that participants don't see the distractors as plausible and increases the odds of someone guessing the correct answer.

Results

With the difficulty index and distractor pattern analysis data, you can make any necessary changes to the test items or distractors you created. After making the changes, roll out your job application knowledge test, knowing that the data you collect measures Level 2 learning and predicts Level 3 job application.

Resources

Phillips, K. 2020. "Evaluate Learning With Predictive Learning Analytics." TD at Work. ATD Press.

Shrock, S. and W. Coscarelli. 2007. Criterion-Referenced Test Development, 3rd ed. Pfeiffer.

Checklist: Calculate a Test Item's Difficulty Index Score

• Count how many people answered test question A. Let's assume 20 people answered it.

• Count how many people correctly answered test question A. In this example, 12 people did.

• Divide the number of people who answered the question correctly by the number of people who answered the test question. This example gives you 0.60 (12 divided by 20).

• Multiply the resulting number by 100. In this case, the difficulty index score is 60 percent, making it a good test question because it falls between 30 percent and 70 percent.