What People Get “Wrong” About Assessment - Part 1

Content

Whether you use the terms assessments, tests, quizzes, knowledge checks, or anything else, trying to figure out what someone knows and can do is likely to be a tricky process. It can also be fraught with anxiety for both the test taker and the test maker. After all, who doesn’t remember feeling terrified while using a No. 2 pencil to fill in tiny ovals on a scantron sheet, believing that your entire future might depend on whether C is really the right answer? And is the feeling really all that different in the workforce if the stakes are potentially getting a promotion or even just keeping your job?

Whether you use the terms assessments, tests, quizzes, knowledge checks, or anything else, trying to figure out what someone knows and can do is likely to be a tricky process. It can also be fraught with anxiety for both the test taker and the test maker. After all, who doesn’t remember feeling terrified while using a No. 2 pencil to fill in tiny ovals on a scantron sheet, believing that your entire future might depend on whether C is really the right answer? And is the feeling really all that different in the workforce if the stakes are potentially getting a promotion or even just keeping your job?

Content

When I took all the big, standardized tests back in the day (no comment on how long ago “the day” was because my kids already gleefully point out how old I am), it never once crossed my mind that those tests were designed and written by actual human beings. In fact, I never gave a single thought about what went into developing a valid and reliable assessment until I was hired at Educational Testing Service in 2006, and suddenly I was the one writing questions for the SAT and the GRE. Now that I’m exploring the talent development field and corporate L&D, I started this blog to share some of what I learned in my nearly 20 years of working on some of the highest-profile assessments on the planet.

When I took all the big, standardized tests back in the day (no comment on how long ago “the day” was because my kids already gleefully point out how old I am), it never once crossed my mind that those tests were designed and written by actual human beings. In fact, I never gave a single thought about what went into developing a valid and reliable assessment until I was hired at Educational Testing Service in 2006, and suddenly I was the one writing questions for the SAT and the GRE. Now that I’m exploring the talent development field and corporate L&D, I started this blog to share some of what I learned in my nearly 20 years of working on some of the highest-profile assessments on the planet.

Content

And what better place to begin than by focusing on the things we often get “wrong” when designing and developing assessments? Yes, the word “wrong” is certainly loaded when talking about tests, which is why I’m using those scare quotes. The key point I would like to make is that there are many, MANY ways to do assessment properly, and there are some things to be aware of so that you do not end up measuring something other than what you think you are measuring.

And what better place to begin than by focusing on the things we often get “wrong” when designing and developing assessments? Yes, the word “wrong” is certainly loaded when talking about tests, which is why I’m using those scare quotes. The key point I would like to make is that there are many, MANY ways to do assessment properly, and there are some things to be aware of so that you do not end up measuring something other than what you think you are measuring.

What People Get Wrong

Content

In fact, the first thing that people tend to get wrong about assessment is NOT being clear about what they are trying to measure. If I don’t know what claims I am trying to make about someone who does well on my assessment, then I will not be able to design a clear and valid test that gathers real evidence to support that claim.

In fact, the first thing that people tend to get wrong about assessment is NOT being clear about what they are trying to measure. If I don’t know what claims I am trying to make about someone who does well on my assessment, then I will not be able to design a clear and valid test that gathers real evidence to support that claim.

Content

Take, for example, an assessment given at the end of a managerial training session. What claims do I want to make about someone who aces that particular test? Does doing well on the test mean that the person is ready to be promoted into a managerial role? Or does it simply mean the person has a basic understanding of a particular managerial approach? Does it mean the person is ready to have difficult conversations with employees? Or maybe it means the person understands my company’s specific process for conducting and documenting year-end reviews?

Take, for example, an assessment given at the end of a managerial training session. What claims do I want to make about someone who aces that particular test? Does doing well on the test mean that the person is ready to be promoted into a managerial role? Or does it simply mean the person has a basic understanding of a particular managerial approach? Does it mean the person is ready to have difficult conversations with employees? Or maybe it means the person understands my company’s specific process for conducting and documenting year-end reviews?

Content

You can see that the design of my assessment is likely to be very different depending on the specific claim(s) I am trying to make about the people taking it. If I am assessing whether or not someone can follow a process, I might give a task-based assessment that asks them to actually do some of the key tasks in the process. If I am assessing knowledge, I am more likely to ask questions (maybe even dreaded multiple choice questions) or work with case studies.

You can see that the design of my assessment is likely to be very different depending on the specific claim(s) I am trying to make about the people taking it. If I am assessing whether or not someone can follow a process, I might give a task-based assessment that asks them to actually do some of the key tasks in the process. If I am assessing knowledge, I am more likely to ask questions (maybe even dreaded multiple choice questions) or work with case studies.

Content

A key mistake many people make when designing an assessment is simply assuming they know what claim(s) they are collecting evidence to support without actually getting specific . Hey, it was training in managerial skills, so we are assessing managerial know-how. Right? Isn’t that enough? Maybe…but you are more likely to get useful information when you plan things out from the beginning and when you are very clear about what you want to know.

A key mistake many people make when designing an assessment is simply assuming they know what claim(s) they are collecting evidence to support without actually getting specific. Hey, it was training in managerial skills, so we are assessing managerial know-how. Right? Isn’t that enough? Maybe…but you are more likely to get useful information when you plan things out from the beginning and when you are very clear about what you want to know.

Content

Researchers at ETS detailed this approach to assessment and called it Evidence-Centered Design (ECD). If you find this sort of thing intriguing and realize to your chagrin that you want to learn more, congratulations! You’ve been bitten by the assessment bug! Fortunately, there are many resources out there that you can delve into. A good place to start is this 2004 paper by Robert Mislevy, Russell Almond, and Janice Lukas , outlining the basic tenets of ECD.

Researchers at ETS detailed this approach to assessment and called it Evidence-Centered Design (ECD). If you find this sort of thing intriguing and realize to your chagrin that you want to learn more, congratulations! You’ve been bitten by the assessment bug! Fortunately, there are many resources out there that you can delve into. A good place to start is this 2004 paper by Robert Mislevy, Russell Almond, and Janice Lukas, outlining the basic tenets of ECD.

Content

To put it in a nutshell, Mislevy, Almond, and Lukas describe the process of ECD in this way:

To put it in a nutshell, Mislevy, Almond, and Lukas describe the process of ECD in this way:

Content

One cannot simply construct ‘good tasks’ in isolation … and hope that someone down the line will figure out ‘how to score it.’ One must design a complex assessment from the very start around the inferences one wants to make, the observations one needs to ground them, the situations that will evoke those observations, and the chain of reasoning that connects them. (2)

One cannot simply construct ‘good tasks’ in isolation … and hope that someone down the line will figure out ‘how to score it.’ One must design a complex assessment from the very start around the inferences one wants to make, the observations one needs to ground them, the situations that will evoke those observations, and the chain of reasoning that connects them. (2)

What To Do Instead

Content

Breaking it down, you have to start, as I said, with the inferences or claims you want to make about the test takers. Once you are clear on what those claims are, you can then start to figure out what observations you might be able to make to give you evidence to support those claims. This is the stage of figuring out what kinds of test questions or “situations” will allow you to observe what you need to observe in order to have evidence that the person taking your test has particular knowledge, skills, or abilities.

Breaking it down, you have to start, as I said, with the inferences or claims you want to make about the test takers. Once you are clear on what those claims are, you can then start to figure out what observations you might be able to make to give you evidence to support those claims. This is the stage of figuring out what kinds of test questions or “situations” will allow you to observe what you need to observe in order to have evidence that the person taking your test has particular knowledge, skills, or abilities.

Content

Doing this work allows you to keep track of the “chain of reasoning” that links all the parts of your assessment together. For a high-stakes assessment, that chain of reasoning should be very well documented. In a typical L&D context, you may not have to do the kind of documentation required for the SAT, but you should at least be clear in your own mind how the evidence you are collecting (the results of the assessment) provides solid support for the claims you are trying to make.

Doing this work allows you to keep track of the “chain of reasoning” that links all the parts of your assessment together. For a high-stakes assessment, that chain of reasoning should be very well documented. In a typical L&D context, you may not have to do the kind of documentation required for the SAT, but you should at least be clear in your own mind how the evidence you are collecting (the results of the assessment) provides solid support for the claims you are trying to make.

Content

In future blog posts, I plan on delving into other ways that people can get assessment “wrong,” looking at things like issues with the wording of specific kinds of test questions, specific problems we have when trying to score assessments (particularly those that are not multiple choice), and issues of bias and sensitivity. If you have assessment issues that you would like me to address—maybe mistakes you’ve made or seen other people make, or just concerns you’ve had—I would love to hear from you. Feel free to reach out to me at [email protected] . I look forward to having this conversation with all of you!

In future blog posts, I plan on delving into other ways that people can get assessment “wrong,” looking at things like issues with the wording of specific kinds of test questions, specific problems we have when trying to score assessments (particularly those that are not multiple choice), and issues of bias and sensitivity. If you have assessment issues that you would like me to address—maybe mistakes you’ve made or seen other people make, or just concerns you’ve had—I would love to hear from you. Feel free to reach out to me at [email protected]. I look forward to having this conversation with all of you!