A Question of Skill

Compare these two questions on communication techniques to de-escalate conflicts in the workplace—which one is more effective at assessing whether learners can apply the skills on the job?

Question 1. What is the primary purpose of paraphrasing in conflict resolution?

A. Reframing the issue in more positive terms

B. Ensuring understanding and demonstrating active listening

C. Summarizing the conversation before moving to the next topic

Question 2. You're a manager working to resolve a conflict between two of your employees. One of your team members, Phil, says: "I'm going to lose it if this favoritism in project assignments doesn't get fixed! Lucia always gets the best projects, while I'm stuck with the routine tasks. I've been here longer, but I keep getting overlooked. I'm starting to wonder if there's any point in trying to advance at this company." What's the best response to de-escalate the conflict?

A. "That's not true, Phil. We distribute projects based on skills, not favoritism. You need to be more patient."

B. "It sounds like you're feeling frustrated because you believe the project assignments aren't equitable, and it's making you question your future here. Is that correct?"

C. "I see. Have you considered that Lucia may be better suited for those high-profile assignments? Maybe you should focus on improving your skills."

The first question isn't bad; it follows many of the guidelines for writing effective multiple-choice questions. However, it's abstract and generic. In contrast, the second question provides relevant context and specific details. Rather than assessing whether people can recall a fact from the training course, the second question checks whether they can apply it in a workplace situation.

One-question mini-scenarios like question 2 are a way to enhance your assessments. A one-question mini-scenario is a multiple-choice question where learners respond to a brief scenario or description of a situation. The question focuses on decision making or application in that scenario.

One-question mini-scenarios differ from other forms of scenario-based learning by their length. The scenarios distill material to a single question and a single action for the learner to take. Like other forms of scenario-based learning, the mini-scenarios provide context to make the assessment realistic and relevant to the specific audience and work environment.

Mini-scenarios focus on decision making within relevant contexts, higher-level thinking, and skills application. Plus, one-question mini-scenarios are relatively easy to implement in virtually any training setting.

Problems with traditional multiple-choice questions

Multiple-choice questions are often abstract and disconnected from real-world application. They also often measure recall of facts or understanding of definitions rather than higher-level thinking.

Let's look at another example question. This one is an assessment question on labor practices for retail managers.

Question 3. Under the Fair Labor Standards Act, what are the thresholds for when overtime pay typically begins for nonexempt employees?

A. 8 hours in a single workday

B. 40 hours in a workweek

C. 80 hours in a two-week pay period

That is a typical question for compliance training. It's specific to the regulation and addresses a common misconception. However, it focuses on remembering the rule rather than applying the rule at work. It's not a terrible question, but it's abstract.

Mini-scenarios make the abstract concrete, giving learners opportunities to apply general conceptual information from training in specific situations. Further, mini-scenarios don't require a significant amount of additional writing—just a few more sentences—to generate.

Question 4. One of your retail associates, Jasper, has a question about reporting the time he worked. "I worked 10 hours yesterday since I covered for Lily's shift too. How do I put that on my timecard? It's eight hours of regular pay plus two hours of overtime, right?" What is the best response to Jasper that follows Fair Labor Standards Act guidelines?

A. "Yes, that's right. Overtime starts for anything over eight hours in a day, so that's eight hours of regular time plus two hours of overtime."

B. "How many total hours have you worked this week? Overtime only applies if you work more than 40 hours in a week."

C. "No, you should report it all as regular time. Overtime pay is for when you work more than 80 hours in a two-week pay period."

Question 4 addresses the same topic as question 3, but it's concrete and specific to the work environment. Question 3 asks whether the learner can remember the rule; question 4 asks whether the learner can apply the rule in a specific scenario. The three choices still address the same points in the revised version: the correct answer plus two common misconceptions as distractors (wrong answers). While the original question assesses low-level thinking, the revised one shifts the focus to higher-level thinking.

Writing effective mini-scenario questions

Crafting a one-question mini-scenario starts with the learning objectives. Assessment questions should align with the objectives.

For example, suppose you're creating a training program on conflict resolution for managers. If one of the learning objectives is "Apply effective communication techniques to de-escalate tensions and promote constructive dialogue," then the assessment should measure application, not just recall.

That said, an effective one-question mini-scenario contains several elements. First, a brief scenario introduces the situation. The scenario should contain concrete details such as names or roles. Specific details help the scenario feel more relevant to your audience. The scenario should contain just enough context to enable learners to decide and select the right option.

After the scenario, ask a question. Avoid asking "What would you do?" because what people would do isn't necessarily the best answer. A better question structure is: "What is the best response, according to the principles the training course covered?" That makes it explicit that you're tying the assessment to the training content.

A one-question mini-scenario also needs plausible alternatives. The distractors should include common mistakes or misconceptions that your audience may have.

Creating good distractors requires understanding your audience, their work environment, and the kinds of mistakes they make. During your analysis when developing a training program, investigate common mistakes and problems. To write effective distractors in assessment questions, it's not enough to know the best way to do things—you need to understand where people go wrong.

Sometimes, you can use partially correct options as distractors. For example, in question 2, the first option states a fact about how the team distributes projects fairly based on skills. That may be a good counterargument to Phil's complaints about favoritism, but it fails to acknowledge and de-escalate his emotions first. That distractor also continues by telling Phil to be more patient, which could escalate the situation. The mix of effective and ineffective responses makes it a plausible distractor. It's partially correct but not as effective as the second option.

The best questions and alternatives in mini-scenarios are related to actions, responses, and behaviors rather than abstract information. Be careful of setting up a good scenario but following it with a question about information rather than decision making. For instance, question 2 could be "What type of communication skill could a manager use to de-escalate this conflict?" That would make it a question about categorization and comprehension rather than application without changing the introductory scenario.

To keep mini-scenarios succinct, avoid including extraneous character backstory. Adding a lot of background is more distracting than helpful. Also, don't make the scenario introduction too long. If you find yourself writing more than a few sentences, think about whether everything in the scenario is necessary.

Converting questions to mini-scenarios

The process of turning traditional multiple-choice questions into mini-scenario questions likewise starts with the learning objectives. Keeping in mind the learning objective of "Apply effective communication techniques to de-escalate tensions and promote constructive dialogue," let's start with an example of a multiple-choice question measuring comprehension.

Question 5. Which communication technique encourages more detailed responses and deeper understanding during conflicts?

A. Paraphrasing

B. Reframing

C. Open-ended questions

That question relates to the learning objective; it clearly references communication techniques. If the course covers those techniques, then the question would measure whether learners comprehend the benefits of open-ended questions.

However, that doesn't measure application. Remembering a definition or understanding the benefits of a technique isn't the same as being able to use it. To measure application, create a scenario in which learners must apply the technique.

To rewrite the question as a scenario, start by thinking about a situation in which your audience would use the information. For conflict resolution, that means some sort of dispute. For an audience of managers, a relevant scenario is a conflict between two direct reports.

Sticking with the topic of open-ended questions, the scenario should include a problem where the manager asks an open-ended question to fully understand the situation before proceeding with other techniques.

Question 6. Jordan, one of your employees, says, "I'm frustrated with the way Elias has been handling our joint project. He constantly changes the plans after we agreed on them, and it's causing delays and confusion. Just yesterday, he completely revamped the presentation we were working on without consulting me first." What's the best way to respond, according to the principles in this training program?

That scenario sets up a conflict to resolve, and it gives the learner an opportunity to apply what they learned in training.

Now, this question needs choices. The right answer is an open-ended question to encourage a detailed response and deeper understanding.

Correct answer: "I see. Can you tell me more about how these changes have affected the project timeline and your work?"

The distractors should be plausible mistakes. What might your audience do wrong in that situation?

Distractor 1: "I hear your frustration. Elias needs to learn to stick to the plan. I'll tell him to stop making changes without your approval."

That distractor acknowledges Jordan's frustration, but it immediately takes their side without taking the time to understand the situation or Elias's perspective. The distractor could appeal to a manager's desire to fix a problem quickly, a common error.

Distractor 2: "It sounds like there's been a misunderstanding. Why don't you try to be more assertive when working with Elias?"

Reframing the situation as a misunderstanding may be helpful, but that response puts the onus on Jordan to change their behavior without addressing Elias's actions. Jordan may perceive the response as dismissing their concerns. That could be another common mistake, which makes it a good distractor. It also includes a question, which makes it similar to the right answer. Thus, learners must understand what an effective open-ended question sounds like to differentiate it from the question in distractor 2.

Putting together the scenario question, correct answer, and distractors, now you have a mini-scenario question that measures application. It aligns with the learning objective to provide a realistic situation in which a manager can apply the techniques they learned in training. It requires deeper thought than only remembering definitions or comprehending general rules for why communication techniques are helpful in resolving conflicts.

Note that, while you can convert many traditional multiple-choice questions to mini-scenario questions, you should not do so for all questions. Consider the following, which is a typical type of query in many compliance training assessments.

Question 7. In what year did the Fair Labor Standards Act become a law?

A. 1928

B. 1938

C. 1948

Does that question measure anything meaningful? When would a manager need to know that? If they didn't know the year the law passed, how would it affect their job performance?

Watch out for questions like that one. They don't align to learning objectives, and they don't reflect the skills people need to do their jobs. You shouldn't rewrite such questions as a scenario; rather, replace it with a meaningful question.

One question at a time

One-question mini-scenarios offer a powerful way to enhance assessments by shifting from abstract recall to practical application. I have found that even clients and organizations that are hesitant to use more complex forms of scenarios are willing to try one-question mini-scenarios. Stakeholders can often immediately see the value of such questions that feel more authentic and tailored to their work environment and role. Including one-question mini-scenarios in your assessments can provide you with better insights into what your audience has learned and their decision-making skills.

Make Distractors Tempting

When crafting distractors, instructional designers should make them plausible rather than obviously wrong. In a question about conflict resolution, "Ignore the argument and hope they work it out eventually" is easy to identify as a wrong answer. However, if you can think about what's going on in your audience's heads, you can rewrite that choice to something more tempting. "Give them some space to resolve the conflict on their own" sounds more positive. It provides rationale for ignoring the problem that may mirror a mistake people make on the job. In fact, that answer may be correct in some situations. That makes it a good distractor.

Another tempting distractor is an option that is the right choice but at the wrong time in a process. For example, assume you're training people on a sales cycle that includes qualifying prospects before sending a proposal. Sending a proposal is a correct step in the process, but it's an error if a learner skips the step of qualifying a prospect first.

Making distractors tempting doesn't mean you should write trick questions or make the options deliberately confusing. Those tactics are just as ineffective at assessment in one-question mini-scenarios as they are in traditional multiple-choice questions. The guidelines for writing reliable and valid multiple-choice questions still apply, even when the question includes a scenario.

A Trio of Choices

You may have noticed that the example questions in this article include three choices rather than four or five alternatives. In the past, the premise was that more alternatives were necessary to develop valid assessments and reduce guessing. However, recent research has shown that you only need three options for multiple-choice questions.

For example, a 2005 meta-analysis in Educational Measurement by Michael C. Rodriguez found that "three options are optimal" for most multiple-choice questions. Reducing the number of options from five or four to three doesn't significantly affect the difficulty or reliability of the questions.

Purya Baghaei and Nazila Amrahi found similar results in their 2011 research published in Psychological Test and Assessment Modeling. They tested the effects of randomly removing one distractor (wrong answer) from numerous multiple-choice questions. Their research shows no significant difference in difficulty or several other measures. They also concluded that three options are ideal for most questions.

For assessments in general, the quality of the distractors matters more than the quantity. That makes instructional designers' jobs easier; they only need one correct answer plus two distractors for a total of three choices.