Top
1.800.628.2783
1.800.628.2783
Advertisement
macro of a business survey, with Strongly Agree checked
Insights

Evidence-Based Survey Design: “Strongly Agree” on the Left or Right Side of the Likert Scale?

Wednesday, September 18, 2019

How many survey questionnaires have you completed during your career? Probably quite a few. Some of the survey items you completed likely used the bipolar Likert Scale, with agreement options on one side, disagreement options on the other side, and neutral in the middle. You may also have designed your own survey questionnaires using the Likert Scale or similar styles—for example, satisfaction options on one side and dissatisfaction option on the other.

When using these scales, there are two ways of listing response options—in ascending or descending order—depending upon whether you list the most positive response option on the left or right side of the response scale. In the following survey item, for example, which order of the Likert Scale options would you use? Does it matter?

I feel valued at work.
○ Strongly agree ○ Agree ○ Neutral ○ Disagree ○ Strongly disagree (horizontal descending order)

I feel valued at work.
○ Strongly disagree ○ Disagree ○ Neutral ○ Agree ○ Strongly agree (horizontal ascending order)

Research has shown that whether you use a descending or ascending scale, the order of response options can make a difference in the survey results. Before we talk about the effects of these, we need to understand some aspects of human psychology and behavior.

Think about this—you’re ordering food at a restaurant, and the server is telling you several available salad dressing or wine options. You listen and, despite all the choices, find yourself selecting one of the last ones listed. This is called a recency effectpeople tend to select the options that they see or hear at the end of the response option list. The recency effect tends to happen when options are presented orally.

When the response options are written in self-administered survey questionnaires, a few outcomes can occur. The primacy effect happens when survey respondents select the options that are presented at the beginning of the response option list listed horizontally (as shown above). Similarly, when the text is written from left to right, survey respondents tend to select what’s written on the left side of the response scale—what’s known as left-side selection bias. Survey respondents also tend to agree rather than disagree with the statement provided—an outcome known as acquiescence bias or yea-saying bias. This choice is also related to respondents’ desires to select options they think are more socially acceptable (often the positive ones) instead of expressing their true thoughts. Social-desirability bias plays into this scenario. Survey respondents also tend to select options that are simply satisfactory (good enough)— satisficing behavior—to minimize their cognitive effort in completing the survey.

When given written self-administered survey items with response options presented horizontally, survey respondents usually select among the ones that:

  • they read first on the left side (primacy effect and left-side selection bias)
  • indicate agreement, which is more socially acceptable and does not require too much thinking to decide (acquiescence bias, social-desirability bias, and satisficing).

So, what would happen when you use the Likert or a Likert-type scale ascending or descending response options? Research has shown mixed results:

1. Descending-ordered response options result in statistically significantly more positive ratings.
2. Different response orders do not make a statistically significant difference in survey ratings.

Between the two, however, the first type of results seem to be more frequent. Why could that be?

As shown in Table 1, the left-side selection bias may apply to ascending- and descending-ordered response scales. When acquiescence bias, social desirability bias, and satisficing behavior are applied to descending-ordered responses scales, you likely see more positive average survey ratings.

Chuyng-Possible Effects of Ascending-Ordered and Descending-Ordered Likert Scales.gif
With that in mind, also think about this—if you were to develop survey items with an 11-point scale to rate, for example, clarity in writing, which one of the following response scales should you be concerned about for inflated average scores due to response order effects?

1. Clear 10 9 8 7 6 5 4 3 2 1 0 Unclear
2. Clear 0 1 2 3 4 5 6 7 8 9 10 Unclear
3. Unclear 10 9 8 7 6 5 4 3 2 1 0 Clear
4. Unclear 0 1 2 3 4 5 6 7 8 9 10 Clear

Research found that the first one that puts positive wording (“Clear”) and the highest numerical value (“10”) on the left side resulted in the highest average score.

Some research, though, showed no statistically significant differences between the ascending- and descending-ordered response scales on the average survey ratings.

Also, what if you present Likert Scale options (as shown below) or other Likert-type scale response options vertically? Will you see top selection bias (similar to left-side selection bias)?

I feel valued at work.

○ Strongly agree
○ Agree
○ Neutral
○ Disagree
○ Strongly disagree

(vertical descending order)

I feel valued at work.

○ Strongly disagree
○ Disagree
○ Neutral
○ Agree
○ Strongly agree

(vertical ascending order)

Again, research shows mixed results. Some studies have not clearly shown top selection bias from vertically presented response options, but some did.

What does this all mean to you as a practitioner when designing your survey items with Likert-type response scales? You intend to use your survey data as part of your evidence-based practice; therefore, it is important that you collect data that is as accurate as possible. Now that you know descending-ordered response scales can give you inflated, inaccurate data, here are some suggestions:

  • If you suspect that your survey items may be prone to showing inflated data when using descending-ordered response scales, use ascending-ordered response scales.
  • Provide clear directions about the order of response options because some survey respondents may use the “first-is-best” heuristic method and assume the far-left option is the most positive option without carefully reading it.
  • If you choose to use descending-ordered response scales, you may want to make your survey questionnaire short because fatigued survey respondents (due to lengthy survey items) are more prone to response-order effects.
About the Author

Yonnie Chyung, EdD, is a professor and associate chair of the Organizational Performance and Workplace Learning department at Boise State University. She teaches graduate courses on program evaluation, quantitative research, and survey design. She is the author of 10-Step Evaluation for Training and Performance Improvement (Sage 2019) and Foundations of Instructional and Performance Technology (HRD Press 2008). Yonnie provides consulting to organizations to perform statistical analysis on their organizational data and conduct program evaluations, often involving students in her research and consulting projects. Recently, she has been developing evidence-based survey design principles for training and performance improvement practitioners.

About the Author

Megan Miller recently graduated from Boise State University's Organizational Performance and Workplace Learning program. She has a passion for using evaluation, needs assessment, and instructional design to support nonprofit organizations. For her master’s thesis, Megan worked with a local nonprofit to evaluate, understand, and build their evaluation capacity. During her master's

program, Megan worked as a graduate assistant with Dr. Chyung working on her team to research and develop articles on evidence-based survey design. Megan has also had the opportunity to present these findings at a Boise State’s Graduate Showcase, International Society for Performance Improvement Annual Conference, and ATD Treasure Valley chapter.

2 Comments
Sign In to Post a Comment
Or you could reject Likert scales altogether and follow the approach recommended by Will Thalheimer in Performance-Focused Smilesheets: A Radical Rethinking of a Dangerous Art form. See https://www.td.org/magazines/td-magazine/its-time-for-a-new-type-of-smile-sheet
Hi Deanna, we certainly have several response scale options including verbal descriptor scales, numerical rating scales (including semantic differential), sliders, etc. I think it is important to select a type of response scale depending on the type of data we need and the way we plan to analyze the data - For example, will we compare pre/post or multi-groups' results? How? Will we reply on percentages of scale options selected, or use means, standard deviations, and other statistical analyses?
Sorry! Something went wrong on our end. Please try again later.
Sorry! Something went wrong on our end. Please try again later.
Sorry! Something went wrong on our end. Please try again later.