ATD Blog
Science of Learning 101: Can We Really Assess Learning?
Fri Feb 06 2015
Content
This is the second post of the Science of Learning 101 series. I am writing this for L&D practitioners and very much want your comments. Is this helpful to you? Does it answer your questions? Your comments can change the course of this series.
This is the second post of the Science of Learning 101 series. I am writing this for L&D practitioners and very much want your comments. Is this helpful to you? Does it answer your questions? Your comments can change the course of this series.
Content
In order to assess (measure) something, we need to know what we want to measure and how to measure it. For example, there are specific ways to measure points scored in different team sports, how fast different objects are moving, and the amount of liquids, solids, and weights used in cooking.
In order to assess (measure) something, we need to know what we want to measure and how to measure it. For example, there are specific ways to measure points scored in different team sports, how fast different objects are moving, and the amount of liquids, solids, and weights used in cooking.
Content
How do we measure learning? In last month’s post , I defined learning as “changes in behavior that result from experience.”
How do we measure learning? In last month’s post, I defined learning as “changes in behavior that result from experience.”
Content
To assess learning, we need to know how to measure the changes in behavior that are most important to us. For example, what do you measure when you need to assess the learning experience of a class on using Google mail? One likely measurement could be does the learner now know how to sign up for and use a Google mail account to send and receive email.
To assess learning, we need to know how to measure the changes in behavior that are most important to us. For example, what do you measure when you need to assess the learning experience of a class on using Google mail? One likely measurement could be does the learner now know how to sign up for and use a Google mail account to send and receive email.
Content
People can and do learn on their own. All. The. Time. But when we want to help people learn, we build instruction—the creation of special environments specifically to promote learning. So the obvious reason for assessing learning is to see if the special environment we built changed what the learner is doing as a result of what we did .
People can and do learn on their own. All. The. Time. But when we want to help people learn, we build instruction—the creation of special environments specifically to promote learning. So the obvious reason for assessing learning is to see if the special environment we built changed what the learner is doing as a result of what we did.
Content
Eek! Research shows that we don’t tend to find out if the training had an impact on what people are doing on the job or in the real world. Rather, we tend to only assess what they can recall from the training. That, my friends, means that we are not regularly doing what most professionals do: assess their work. As you will see by my discussion below, this is not rocket science.
Eek! Research shows that we don’t tend to find out if the training had an impact on what people are doing on the job or in the real world. Rather, we tend to only assess what they can recall from the training. That, my friends, means that we are not regularly doing what most professionals do: assess their work. As you will see by my discussion below, this is not rocket science.
Content
Different Ways to Assess Learning
Different Ways to Assess Learning
Content
Testing with recall questions about the content really isn’t much of an assessment at all. Here are a few examples of real-life learning assessments.
Testing with recall questions about the content really isn’t much of an assessment at all. Here are a few examples of real-life learning assessments.
Content
Chen demonstrates he is a good candidate to move into the vacant account executive position by his sophisticated product knowledge and excellent dealings with clients.
Chen demonstrates he is a good candidate to move into the vacant account executive position by his sophisticated product knowledge and excellent dealings with clients.
Content
Nola creates unique jewelry out of twisted strands of sterling silver. Her students anxiously await her new classes.
Nola creates unique jewelry out of twisted strands of sterling silver. Her students anxiously await her new classes.
Content
Kia helps Penn with the more difficult statistics coursework for their classes. Both are hoping to move into more advanced and lucrative positions within the company.
Kia helps Penn with the more difficult statistics coursework for their classes. Both are hoping to move into more advanced and lucrative positions within the company.
Content
All of these people learned their craft through informal and formal learning. Stop for a moment and consider what they might have done to improve their expertise.
All of these people learned their craft through informal and formal learning. Stop for a moment and consider what they might have done to improve their expertise.
Content
Traditionally, educational classroom assessment provides grades and placements, helps determine where teachers need to give additional support, finds where to address misunderstandings, and targets feedback.
Traditionally, educational classroom assessment provides grades and placements, helps determine where teachers need to give additional support, finds where to address misunderstandings, and targets feedback.
Content
Too often, in traditional corporate classroom and online learning, we don’t determine if people need more support or resolve misunderstandings—so it may become difficult for people to continue learning. Instructional feedback is often too shallow (answer: correct/incorrect). Likewise, too many designers don’t realize that feedback is also meant to be instruction!
Too often, in traditional corporate classroom and online learning, we don’t determine if people need more support or resolve misunderstandings—so it may become difficult for people to continue learning. Instructional feedback is often too shallow (answer: correct/incorrect). Likewise, too many designers don’t realize that feedback is also meant to be instruction!
Content
Direct and Indirect Inferences About Learning
Direct and Indirect Inferences About Learning
Content
We make inferences (a conclusion made on the basis of evidence and reasoning, an educated guess) about learning by collecting direct or indirect evidence.
We make inferences (a conclusion made on the basis of evidence and reasoning, an educated guess) about learning by collecting direct or indirect evidence.
Content
Direct evidence includes samples of what people do, which tends to be the strongest evidence. Chen’s answers to product knowledge questions are direct evidence, as are Kia’s statistics skills. We can’t measure everything directly, such as perceptions or feelings.
Direct evidence includes samples of what people do, which tends to be the strongest evidence. Chen’s answers to product knowledge questions are direct evidence, as are Kia’s statistics skills. We can’t measure everything directly, such as perceptions or feelings.
Content
Indirect evidence includes perceptions and rankings. These can come from supervisors, peers, customers, the person, and others. For example, we might ask Kia if Penn seems to be struggling with statistics. A survey of Nola’s teaching skills by her students would be indirect evidence (observations of her classes would be more direct). Indirect evidence tends to be less strong than direct evidence because it’s more subjective.
Indirect evidence includes perceptions and rankings. These can come from supervisors, peers, customers, the person, and others. For example, we might ask Kia if Penn seems to be struggling with statistics. A survey of Nola’s teaching skills by her students would be indirect evidence (observations of her classes would be more direct). Indirect evidence tends to be less strong than direct evidence because it’s more subjective.
Content
An ideal assessment might combine direct and indirect measures. We call this “triangulation” of assessment evidence. Table 1 below provides a list of possible direct and indirect evidence of learning. There are more types of evidence but these are the main types for training.
An ideal assessment might combine direct and indirect measures. We call this “triangulation” of assessment evidence. Table 1 below provides a list of possible direct and indirect evidence of learning. There are more types of evidence but these are the main types for training.
Content
Table 1: Direct and Indirect Evidence of Learning
Table 1: Direct and Indirect Evidence of Learning
Content | Content |
Content Direct Assessment Direct Assessment | Content Indirect Assessment Indirect Assessment |
Content Scenario question results Scenario question results Content Observations Observations Content Actual performance (simulation) Actual performance (simulation) Content Actual performance (on the job) Actual performance (on the job) Content Work products (such as reports) Work products (such as reports) | Content Checklist of desired behaviors: Supervisor, self, peers, and customers Checklist of desired behaviors: Supervisor, self, peers, and customers Content Supervisor, self, peer, and customers rating Supervisor, self, peer, and customers rating Content Interviews with supervisor, self, peer, and customers Interviews with supervisor, self, peer, and customers Content Graduation from training programs Graduation from training programs |
Content
| Content |
Content
Measuring learning is difficult because we have to first agree on what exactly we are measuring. And we could spend hours (or more) arguing even the definition of learning! One thing is certain: we are not measuring direct evidence such as speed or weight; we are measuring the outcome of the evidence, something that stands in for the evidence.
Measuring learning is difficult because we have to first agree on what exactly we are measuring. And we could spend hours (or more) arguing even the definition of learning! One thing is certain: we are not measuring direct evidence such as speed or weight; we are measuring the outcome of the evidence, something that stands in for the evidence.
Content
Triangulating data (by looking at direct and indirect evidence) is often worthwhile. It increases the chance that you are getting a decent picture of what is happening. It’s really hard to reduce this picture to a single number unless you are statistically proficient, and even then, you are reducing something or an initiative to a number. I’m not usually a fan of that because you often lose sight of what that number really represents. I recommend you read Dr. Brinkerhoff’s Success Case Method (SCM) for evaluating the success of a training initiative.
Triangulating data (by looking at direct and indirect evidence) is often worthwhile. It increases the chance that you are getting a decent picture of what is happening. It’s really hard to reduce this picture to a single number unless you are statistically proficient, and even then, you are reducing something or an initiative to a number. I’m not usually a fan of that because you often lose sight of what that number really represents. I recommend you read Dr. Brinkerhoff’s Success Case Method (SCM) for evaluating the success of a training initiative.
Content
What Is the Purpose of Assessment?
What Is the Purpose of Assessment?
Content
We may believe that assessment serves to tell the one person taking instruction how they are doing, but that’s only one of many reasons for assessment. Next month I’ll discuss other reasons for assessment—and why leaving them out actually leaves L&D practitioners in peril.
We may believe that assessment serves to tell the one person taking instruction how they are doing, but that’s only one of many reasons for assessment. Next month I’ll discuss other reasons for assessment—and why leaving them out actually leaves L&D practitioners in peril.
Content
References
References
Content
Brinkerhoff, R. O. (2006). Telling training's story: Evaluation made simple, credible, and effective. Berrett-Koehler Publishers.
Brinkerhoff, R. O. (2006). Telling training's story: Evaluation made simple, credible, and effective. Berrett-Koehler Publishers.
Content
De Houwer, J., Barnes-Holmes, D., and Moors, A. (2013). What is learning? On the nature and merits of a functional definition of learning. Psychonomic bulletin & review. http://www.liplab.ugent.be/pdf/learningpreprint.pdf
De Houwer, J., Barnes-Holmes, D., and Moors, A. (2013). What is learning? On the nature and merits of a functional definition of learning. Psychonomic bulletin & review. http://www.liplab.ugent.be/pdf/learningpreprint.pdf
Content
Mayer, R. E. (2011). Applying the science of learning. Upper Saddle River, NJ: Pearson.
Mayer, R. E. (2011). Applying the science of learning. Upper Saddle River, NJ: Pearson.
Content
Shank, P. (January 8, 2015.) Science of Learning 101: What’s the Point of Instruction? ATD Science of Learning Blog. https://www.td.org/content/atd-blog/science-of-learning-101-whats-the-point-of-instruction
Shank, P. (January 8, 2015.) Science of Learning 101: What’s the Point of Instruction? ATD Science of Learning Blog. https://www.td.org/content/atd-blog/science-of-learning-101-whats-the-point-of-instruction
Content