Surveys are a big part of a talent development professional’s life. We’re constantly seeking out data to measure everything from training effectiveness to employee engagement. We develop surveys hoping that the results will give us some idea of where to focus our resources to make the biggest organizational impact.
We know a lot of what to do right when it comes to surveys - we spend hours finding the right survey tool, thinking about timing, and contemplating which of Kirkpatrick’s 4 Levels we are trying to evaluate. We proof-read, spell check, and beta test our survey tool, but in all that time there is one simple element that I see missed over and over again - double-loading questions.
What are double-loading questions?
Double-loading questions are questions that address two domains but only allow for one response. For example, “I was satisfied with the pace and material presented in this course.” Seems like a standard survey question, right? But let’s take a closer look....
Why are double-loading questions a problem?
In the example above we are asking people to rate two domains - the pace of the course AND the material presented in the course. What if I was satisfied with the material presented but not with the pace of the course? Given a 5-point likert scale, I can only choose one response.
Now when you, as the talent development professional, review the data how do you know which part of the question is reflected in my response? You don’t. Even worse, some respondents will answer on one domain and some will answer on the other domain. In the end, the data on this question is essentially useless - you don’t know which responses were for which domain. You don’t know if your audience was truly satisfied or dissatisfied with either or satisfied with one and not the other. Just thinking about it is enough to make your head spin!
So how do I fix it?
Luckily, it’s an easy fix. Review your questions and think about their structure - could there be two (or more!) parts that people might want to rate separately?
And there’s a couple tricks you can use. First, every time you see the words “and” or “or” seriously reconsider if it should be two separate questions. Consider those words your warning flags.
Second, look out for lengthy questions. If the question is more than one line, odds are good that you have a double-loading question on your hands.
Neither of these “tricks” guarantees you will find a double-loading question but they are definitely things to keep your eye out for when reviewing your surveys. Once you identify one, the fix is easy - separate the question into two (or more) questions to be sure the data you collect will measure just one construct and tell you EXACTLY where you need to make improvements.
Can’t I just change the question format?
Do you think changing the question to an open-ended or a multiple-response format will ease your double-loading question woes? Ah, not so fast! Even with another question format, how do you know that your respondents gave their answers based on both constructs? Even if they do give responses to both constructs, will it be clear to you which portions of the response are directed at which domains? Most likely not.
But now you know a simple fix to a huge survey development problem and, more importantly, WHY it is such a problem. Don’t feel bad if you’ve been double-loading - I see this every. single. time I take a survey (seriously, without fail) so you are not the only one. Just take action to fix it and remember that keeping your questions simple and straight-forward without those pesky double-loading questions will bring you the best survey results so you know exactly where to focus your efforts.