ATD Blog

Problems with Survey Questions

Wednesday, August 14, 2013

There are two parts to a survey question: the question stem and the scale for response choices. In many situations, entirely too much time is given to deciding which scale to use. In fact, if the stem is poorly written, then the scale means very little.

In 2001, David Murray presented examples of typical problems found in survey questions for Training. Twelve years later, the same problems exist.

Below is his list of statements that represent stems on a Likert-type scale. As you read each statement consider the issues with the statement and the alternative. Then, think about your survey questions. Do they have similar issues?

When writing your survey questions, write them for the respondent. While you may know what you mean, if they don’t, what value does the survey process bring?

Problem Stem


Better Problem Stem

When making assignments, my manager gives clear, achievable goals that are within my control.


This is called barreling. There are three questions in one. Questions should be specific, addressing one issue at a time.

When making assignments, my managers give clear goals.

I am interested in my students.

What does this mean? This very loose question could mean any number of things. Questions should be anchored by behaviors or specific indicators that reflect the intent of the question.

I regularly share information about my students’ progress with their parents.


I would speak more freely if my co-workers didn’t have such long left-hand columns.



If you are familiar with the work of Peter Senge and his right-hand (what people say) and left-hand (what people think about what is being said) columns, you might be able to answer this. Otherwise, the jargon is unclear to respondents.

I would speak more freely if my co-workers were more open about what they are thinking.

The sales of automatic weapons should be banned in order to save human lives.


Asked this way, respondent are likely to answer in the way the question is leading them. But the question is: Do you want respondents to provide you their answer or your answer? This loaded question may get you the answer you want, but not necessarily the answer.

The sales of automatic weapons should be banned.

The instructor is an expert in this field.



Question stems such as this are not unusual; unfortunately, the audience may not have the knowledge they need to answer the question objectively.

The instructor effectively answered all of my questions on this topic.

What is the frequency of the overall interpersonal, informal, and formal communication between the focal group and the targeted semi-autonomous, functionally specialized groups?


Do you understand this? Adding a layer of unnecessary complexity adds no value to the question. Keep questions simple.

What is the frequency of formal communication between the focal group and your group?

I frequently belittle and talk down to my fellow employees.


For those managers reading this, I’m sure you would answer in the negative, even if you do belittle and talk down to your employees.

Members of my work group show respect by eliciting each other’s opinions.

For more from Patti check out her book, Survey Basics.

About the Author

Patti Phillips is president and CEO of the ROI Institute and is the ATD Certification Institute's 2015 CPLP Fellow. Since 1997, she has worked with organizations in more than 60 countries as they demonstrate the value of a variety of programs and projects. Patti serves on the board of the Center for Talent Reporting, as Distinguished Principal Research Fellow for The Conference Board, and as faculty on the UN System Staff College in Turin, Italy.

Patti has written and edited numerous books and articles on the topics of measurement, evaluation, and ROI. Recent publications include Measuring the Success of Leadership Development, Making Human Capital Analytics Work, Measuring the Success of Learning Through Technology, Measuring the Success of Organization Development, and Measuring Leadership Development: Quantify Your Program's Impact and ROI on Organizational Performance.

Be the first to comment
Sign In to Post a Comment
Sorry! Something went wrong on our end. Please try again later.