ATD Blog
What Do You Know: What Should We Include in Feedback?
Wed Mar 29 2017
Content
In my last post about feedback, I discussed how research shows that feedback can improve learning. But it doesn’t always. Hattie and Timperley (2007) explain that feedback helps learning most when it deals with three specific learning problems:
In my last post about feedback, I discussed how research shows that feedback can improve learning. But it doesn’t always. Hattie and Timperley (2007) explain that feedback helps learning most when it deals with three specific learning problems:
Content
Misunderstandings . For example, people may think that saving is important, but that it doesn’t matter when you save. (Wrong! It matters because of compound interest!)
Misunderstandings. For example, people may think that saving is important, but that it doesn’t matter when you save. (Wrong! It matters because of compound interest!)
Content
Forgotten information or strategies . For example: Which is more important? Saving for retirement or paying off debt? (It depends on a number of factors.)
Forgotten information or strategies. For example: Which is more important? Saving for retirement or paying off debt? (It depends on a number of factors.)
Content
Performance difficulties . For example, giving in to “tricks of the trade” may increase what you pay without much benefit (such as glass etching on windshields and extended warrantees on most products).
Performance difficulties. For example, giving in to “tricks of the trade” may increase what you pay without much benefit (such as glass etching on windshields and extended warrantees on most products).
Content
Now that we know the kind of feedback that most improves learning, let’s discuss what research tells us we should include in feedback. But first, review the following question and select the answers you think are correct.
Now that we know the kind of feedback that most improves learning, let’s discuss what research tells us we should include in feedback. But first, review the following question and select the answers you think are correct.
Content
Which of the following is valuable as feedback? Select all that apply.
Which of the following is valuable as feedback? Select all that apply.
Content
A. Whether the answer is correct or incorrect
A. Whether the answer is correct or incorrect
Content
B. (If incorrect) Try again?
B. (If incorrect) Try again?
Content
C. The correct answer
C. The correct answer
Content
D. Hints to improve recall of the correct answer, such as applicable rules and facts
D. Hints to improve recall of the correct answer, such as applicable rules and facts
Content
E. Hints on useful strategies, typical misunderstandings, and sources of typical mistakes
E. Hints on useful strategies, typical misunderstandings, and sources of typical mistakes
Content
F. Analysis of errors made (what is the misunderstanding or source of mistake)
F. Analysis of errors made (what is the misunderstanding or source of mistake)
Content
G. Where training discussed this information H. Worked examples showing how to get to the correct answer, such as steps or rationale
G. Where training discussed this information H. Worked examples showing how to get to the correct answer, such as steps or rationale
Content
I. Interesting tidbits to motivate use of feedback
I. Interesting tidbits to motivate use of feedback
Do We Need Feedback?
Content
Writing proper assessments and feedback is time-consuming. Many designers tell me that they have difficulty with these skills. It’s understandable that people would love to get rid of these tasks. But when these tasks are done well, they add a great deal of value to instruction. (I will write about interesting outcomes from testing soon.)
Writing proper assessments and feedback is time-consuming. Many designers tell me that they have difficulty with these skills. It’s understandable that people would love to get rid of these tasks. But when these tasks are done well, they add a great deal of value to instruction. (I will write about interesting outcomes from testing soon.)
Content
Feedback helps when it offers the right information at the right time and in the right way. In the rest of this post, we’ll discuss what to include in feedback. and a little bit about how to offer it. This information is critical for learning designers and instructors because bad feedback can harm learning. And no feedback leaves people stuck with whatever misunderstandings, forgotten information, and difficulties they have. Most learning builds on prior knowledge, so not making sure that people “get it” is simply not OK.
Feedback helps when it offers the right information at the right time and in the right way. In the rest of this post, we’ll discuss what to include in feedback. and a little bit about how to offer it. This information is critical for learning designers and instructors because bad feedback can harm learning. And no feedback leaves people stuck with whatever misunderstandings, forgotten information, and difficulties they have. Most learning builds on prior knowledge, so not making sure that people “get it” is simply not OK.
Content
So yeah, valuable feedback is kind of important.
So yeah, valuable feedback is kind of important.
What Should Feedback Do?
Content
According to Narciss and Huth (2004), whose chapter in Instructional Design for Multimedia Learning I am discussing in this and the following sections, the most important function of feedback is to help participants understand and control their own learning. In other words, feedback should help people assess what they know, don’t know, and need to know. We can add this to Hattie and Timperley (2007)’s feedback goal of reducing the gap between existing knowledge and skills and needed knowledge and skills.
According to Narciss and Huth (2004), whose chapter in Instructional Design for Multimedia Learning I am discussing in this and the following sections, the most important function of feedback is to help participants understand and control their own learning. In other words, feedback should help people assess what they know, don’t know, and need to know. We can add this to Hattie and Timperley (2007)’s feedback goal of reducing the gap between existing knowledge and skills and needed knowledge and skills.
Content
Narciss and Huth (2004) offer us a way to think about types of feedback that I found quite helpful. The two types are quantitative and qualitative (they used different terms but I’m using terms that I think are easier to understand).
Narciss and Huth (2004) offer us a way to think about types of feedback that I found quite helpful. The two types are quantitative and qualitative (they used different terms but I’m using terms that I think are easier to understand).
Content
Quantitative feedback tells people how many. It supplies numbers or percentages, such as how many correct and incorrect responses, what percentage of answers are correct and incorrect, and what percentage of content mastered.
Quantitative feedback tells people how many. It supplies numbers or percentages, such as how many correct and incorrect responses, what percentage of answers are correct and incorrect, and what percentage of content mastered.
Content
Qualitative feedback tells people why. It supplies information to improve understanding of the topic, task, error, or solution.
Qualitative feedback tells people why. It supplies information to improve understanding of the topic, task, error, or solution.
Content
Table 1 discusses the different content elements we might include in quantitative and qualitative feedback.
Table 1 discusses the different content elements we might include in quantitative and qualitative feedback.
Content
Table 1. The Value of Different Feedback Content
Table 1. The Value of Different Feedback Content
Content | Content | Content |
Content
| Content Feedback Content Feedback Content | Content Valuable? Why? Valuable? Why? |
Content quantitative quantitative | Content Whether the answer is correct or incorrect Whether the answer is correct or incorrect Content Incorrect Incorrect | Content Correct Correct Content or or Content Incorrect Incorrect Content is more valuable to those with incorrect answers. But it isn’t enough information. is more valuable to those with incorrect answers. But it isn’t enough information. |
Content Try again Try again Content That is the incorrect answer. Try again. That is the incorrect answer. Try again. | Content Try again Try again Content too often allows people to guess at another answer or keep trying until they get the correct answer. Unless a different question is presented that tests the same information, this typically isn’t valuable. too often allows people to guess at another answer or keep trying until they get the correct answer. Unless a different question is presented that tests the same information, this typically isn’t valuable. | Content |
Content The correct answer The correct answer Content The correct answer is B. Ask a manager to sign off on the withdrawal. The correct answer is B. Ask a manager to sign off on the withdrawal. | Content Supplying the correct answer tells which answer is correct but leaves out why. Supplying the correct answer tells which answer is correct but leaves out why. | Content |
Content Quantitative feedback content rarely offers adequate information to close knowledge and skills gaps. It doesn’t help people improve their ability to think through their understanding and how to fix it. We can consider most quantitative feedback as insufficient. Quantitative feedback content rarely offers adequate information to close knowledge and skills gaps. It doesn’t help people improve their ability to think through their understanding and how to fix it. We can consider most quantitative feedback as insufficient. | Content | Content |
Content qualitative qualitative | Content Hints to improve recall of the correct answer, such as applicable rules and facts Hints to improve recall of the correct answer, such as applicable rules and facts Content Hint: We discussed two different approaches for deciding what order to pay off debts in module 2. Do you remember the order suggested for those with greater than $20,000 in debt? Hint: We discussed two different approaches for deciding what order to pay off debts in module 2. Do you remember the order suggested for those with greater than $20,000 in debt? | Content The types of qualitative feedback content that can be valuable for helping people do one or both of these: The types of qualitative feedback content that can be valuable for helping people do one or both of these: Content Bridge the gap between current knowledge and skills and needed knowledge and skills. Bridge the gap between current knowledge and skills and needed knowledge and skills. Content Help people assess what they know, don’t know, and need to know. Which qualitative feedback elements to choose depends on which problem the wrong answer reveals: Help people assess what they know, don’t know, and need to know. Which qualitative feedback elements to choose depends on which problem the wrong answer reveals: Content misunderstanding misunderstanding Content forgotten information or strategies forgotten information or strategies Content performance difficulties. performance difficulties. Content It is our job to analyze which of the three types of learning problems the incorrect answer reveals. Then we can tailor feedback content to that specific problem. Too much information risks overwhelming the learner and reduced effort. Too little information risks not fixing the problem. Narciss and Huth (2004) list research findings that help us select the best combination of feedback content. It is our job to analyze which of the three types of learning problems the incorrect answer reveals. Then we can tailor feedback content to that specific problem. Too much information risks overwhelming the learner and reduced effort. Too little information risks not fixing the problem. Narciss and Huth (2004) list research findings that help us select the best combination of feedback content. Content Don’t provide any feedback before people have tried to solve it on their own. When we offer feedback too soon, we often reduce effort and learning. Don’t provide any feedback before people have tried to solve it on their own. When we offer feedback too soon, we often reduce effort and learning. Content Don’t provide the correct response and more information (qualitative) feedback together. When we do this, we often reduce effort and allow very shallow learning. Don’t provide the correct response and more information (qualitative) feedback together. When we do this, we often reduce effort and allow very shallow learning. Content Provide qualitative feedback in manageable chunks. This allows people to take in as much as they can handle before taking in more information. Provide qualitative feedback in manageable chunks. This allows people to take in as much as they can handle before taking in more information. Content Reduce cognitive overload with complex feedback. For example, if you are showing how something works, explain using audio rather than in text. (I’ll explain media effects another time.) Point to the relevant aspects of the screen as needed. Reduce cognitive overload with complex feedback. For example, if you are showing how something works, explain using audio rather than in text. (I’ll explain media effects another time.) Point to the relevant aspects of the screen as needed. Content
|
Content Hints on useful strategies, typical misunderstandings, and sources of typical mistakes Hints on useful strategies, typical misunderstandings, and sources of typical mistakes Content Hint: There are some common mistakes people make when creating a budget. You left a key category out of your list. Do you remember which category you need to add? Hint: There are some common mistakes people make when creating a budget. You left a key category out of your list. Do you remember which category you need to add? | Content | Content |
Content An analysis of errors made (what is the misunderstanding or source of mistake) An analysis of errors made (what is the misunderstanding or source of mistake) Content There are some common mistakes people make when creating a budget. You didn’t add a category for unexpected maintenance. Add it now? There are some common mistakes people make when creating a budget. You didn’t add a category for unexpected maintenance. Add it now? | Content | Content |
Content Where training discussed this information Where training discussed this information Content We discussed information about unexpected maintenance in the fourth section. Would you like to revisit that section now before answering more questions? We discussed information about unexpected maintenance in the fourth section. Would you like to revisit that section now before answering more questions? | Content | Content |
Content Worked examples showing how to get to the correct answer, such as steps or rationale Worked examples showing how to get to the correct answer, such as steps or rationale Content In the section on creating a budget, we offered six examples of budgets for families with different needs and incomes. Would you like to review them now? In the section on creating a budget, we offered six examples of budgets for families with different needs and incomes. Would you like to review them now? | Content | Content |
Content Interesting tidbits to motivate use of feedback Interesting tidbits to motivate use of feedback | Content Don’t include! Extra information is likely to result in overload (extraneous cognitive load). Don’t include! Extra information is likely to result in overload (extraneous cognitive load). | Content |
Content
So what is the answer to the question I posed? Everything except I. To see if you understand this post, I’ll ask the following questions:
So what is the answer to the question I posed? Everything except I. To see if you understand this post, I’ll ask the following questions:
Content
What are the two goals or purposes of feedback?
What are the two goals or purposes of feedback?
Content
Quantitative feedback is insufficient feedback because\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_.
Quantitative feedback is insufficient feedback because\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_.
Content
Qualitative feedback should address \_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_.
Qualitative feedback should address \_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_.
References
Content
Hattie, J., and H. Timperley. 2007. “The Power of Feedback.” Review of Educational Research 77(1): 81-112. www.columbia.edu/~mvp19/ETF/Feedback.pdf
Hattie, J., and H. Timperley. 2007. “The Power of Feedback.” Review of Educational Research 77(1): 81-112. www.columbia.edu/~mvp19/ETF/Feedback.pdf
Content
Kulhavy, R.W. 1977. “Feedback in Written Instruction.” Review of Educational Research 47(1): 211-232. http://journals.sagepub.com/doi/pdf/10.3102/00346543047002211
Kulhavy, R.W. 1977. “Feedback in Written Instruction.” Review of Educational Research 47(1): 211-232. http://journals.sagepub.com/doi/pdf/10.3102/00346543047002211
Content
Narciss, S. 2008. “Feedback Strategies for Interactive Learning Tasks.” In Handbook of Research on Educational Communications and Technology , 3rd edition, edited by J.M. Spector, M.D. Merril, J.J.G. van Merriënboer, and M.P. Driscoll, 125-144. Mahwah, NJ: Lawrence Erlbaum. www.phil.uni-passau.de/fileadmin/dokumente/lehrstuehle/narciss/Narciss\_\_S.\_\_2008\_.\_Feedback\_strategies\_for\_interactive\_learning\_tasks.pdf
Narciss, S. 2008. “Feedback Strategies for Interactive Learning Tasks.” In Handbook of Research on Educational Communications and Technology, 3rd edition, edited by J.M. Spector, M.D. Merril, J.J.G. van Merriënboer, and M.P. Driscoll, 125-144. Mahwah, NJ: Lawrence Erlbaum. www.phil.uni-passau.de/fileadmin/dokumente/lehrstuehle/narciss/Narciss\_\_S.\_\_2008\_.\_Feedback\_strategies\_for\_interactive\_learning\_tasks.pdf
Content
Narciss S., and K. Huth. 2004. “How to Design Informative Tutoring Feedback for Multimedia Learning.” In Instructional Design for Multimedia Learning , edited by H.M. Niegemann, D. Leutner, and R. Brunken, 181-195. Munster, NY: Waxmann. www.studierplatz2000.tu-dresden.de/toolkit/presentations/CD/Literatur/Publikationen/ID\_nahu.pdf
Narciss S., and K. Huth. 2004. “How to Design Informative Tutoring Feedback for Multimedia Learning.” In Instructional Design for Multimedia Learning, edited by H.M. Niegemann, D. Leutner, and R. Brunken, 181-195. Munster, NY: Waxmann. www.studierplatz2000.tu-dresden.de/toolkit/presentations/CD/Literatur/Publikationen/ID\_nahu.pdf