ATD Blog
Problems with Survey Questions
Wed Aug 14 2013
Content
There are two parts to a survey question: the question stem and the scale for response choices. In many situations, entirely too much time is given to deciding which scale to use. In fact, if the stem is poorly written, then the scale means very little.
There are two parts to a survey question: the question stem and the scale for response choices. In many situations, entirely too much time is given to deciding which scale to use. In fact, if the stem is poorly written, then the scale means very little.
Content
In 2001, David Murray presented examples of typical problems found in survey questions for Training . Twelve years later, the same problems exist.
In 2001, David Murray presented examples of typical problems found in survey questions for Training. Twelve years later, the same problems exist.
Content
Below is his list of statements that represent stems on a Likert-type scale. As you read each statement consider the issues with the statement and the alternative. Then, think about your survey questions. Do they have similar issues?
Below is his list of statements that represent stems on a Likert-type scale. As you read each statement consider the issues with the statement and the alternative. Then, think about your survey questions. Do they have similar issues?
Content
When writing your survey questions, write them for the respondent. While you may know what you mean, if they don’t, what value does the survey process bring?
When writing your survey questions, write them for the respondent. While you may know what you mean, if they don’t, what value does the survey process bring?
Content | Content | Content |
Content Problem Stem Problem Stem | Content Issues Issues | Content Better Problem Stem Better Problem Stem |
Content When making assignments, my manager gives clear, achievable goals that are within my control. When making assignments, my manager gives clear, achievable goals that are within my control. | Content This is called barreling. There are three questions in one. Questions should be specific, addressing one issue at a time. This is called barreling. There are three questions in one. Questions should be specific, addressing one issue at a time. | Content When making assignments, my managers give clear goals. When making assignments, my managers give clear goals. |
Content I am interested in my students. I am interested in my students. | Content What does this mean? This very loose question could mean any number of things. Questions should be anchored by behaviors or specific indicators that reflect the intent of the question. What does this mean? This very loose question could mean any number of things. Questions should be anchored by behaviors or specific indicators that reflect the intent of the question. | Content I regularly share information about my students’ progress with their parents. I regularly share information about my students’ progress with their parents. |
Content I would speak more freely if my co-workers didn’t have such long left-hand columns. I would speak more freely if my co-workers didn’t have such long left-hand columns. | Content If you are familiar with the work of Peter Senge and his right-hand (what people say) and left-hand (what people think about what is being said) columns, you might be able to answer this. Otherwise, the jargon is unclear to respondents. If you are familiar with the work of Peter Senge and his right-hand (what people say) and left-hand (what people think about what is being said) columns, you might be able to answer this. Otherwise, the jargon is unclear to respondents. | Content I would speak more freely if my co-workers were more open about what they are thinking. I would speak more freely if my co-workers were more open about what they are thinking. |
Content The sales of automatic weapons should be banned in order to save human lives The sales of automatic weapons should be banned in order to save human lives Content . . Content
| Content Asked this way, respondent are likely to answer in the way the question is leading them. But the question is: Do you want respondents to provide you their answer or your answer? This loaded question may get you the answer you want, but not necessarily the answer. Asked this way, respondent are likely to answer in the way the question is leading them. But the question is: Do you want respondents to provide you their answer or your answer? This loaded question may get you the answer you want, but not necessarily the answer. | Content The sales of automatic weapons should be banned. The sales of automatic weapons should be banned. |
Content The instructor is an expert in this field. The instructor is an expert in this field. | Content Question stems such as this are not unusual; unfortunately, the audience may not have the knowledge they need to answer the question objectively. Question stems such as this are not unusual; unfortunately, the audience may not have the knowledge they need to answer the question objectively. | Content The instructor effectively answered all of my questions on this topic. The instructor effectively answered all of my questions on this topic. |
Content What is the frequency of the overall interpersonal, informal, and formal communication between the focal group and the targeted semi-autonomous, functionally specialized groups? What is the frequency of the overall interpersonal, informal, and formal communication between the focal group and the targeted semi-autonomous, functionally specialized groups? | Content Do you understand this? Adding a layer of unnecessary complexity adds no value to the question. Keep questions simple. Do you understand this? Adding a layer of unnecessary complexity adds no value to the question. Keep questions simple. | Content What is the frequency of formal communication between the focal group and your group? What is the frequency of formal communication between the focal group and your group? |
Content I frequently belittle and talk down to my fellow employees. I frequently belittle and talk down to my fellow employees. | Content For those managers reading this, I’m sure you would answer in the negative, even if you do belittle and talk down to your employees. For those managers reading this, I’m sure you would answer in the negative, even if you do belittle and talk down to your employees. | Content Members of my work group show respect by eliciting each other’s opinions. Members of my work group show respect by eliciting each other’s opinions. |
Content For more from Patti check out her book, For more from Patti check out her book, Content Survey Basics. Survey Basics. |