Jack and I have the opportunity to teach ROI workshops around the world. Audiences include representatives of some of the world’s largest organizations, including those in the private sector, public sector, and social sector. Without a doubt, the most commonly used data collection technique when evaluating programs is the self-administered questionnaire.
Yet, also without a doubt, the top concern of evaluators who collect data via the self-administered questionnaire is the degree to which respondents will lie when responding.
Do you think respondents lie on questionnaires?
Some argue that organization culture causes respondents to lie on surveys. If there is fear in the workplace, there may be some truth to the assumption that participants lie. But, as I tell our groups when we discuss managing survey data across cultures, if there is a culture of lying and you still see a normal curve in your data, you’re probably observe what is “accurate” for that particular group.
Some people argue that participants will lie on a survey because, by our nature, we just don’t tell the truth when it comes to responding to questions. Yet, as described in the book The Wisdom of Crowds, if you get enough people together, you will probably get a relatively accurate accounting of responses to most questions.
Of course, suggesting that participants lie on surveys is somewhat brash. However, participants do respond with less objectivity if little thought has been put into how you can get the most truthful responses. So how can you get “honest” answers on self-administered questionnaires? Here are six approaches that may help.
- First develop good program objectives; then write questions that reflect those objectives. Too often surveys include meaningless questions that result in meaningless responses.
- Include well-written questions on your survey. Vague, complex, poorly written questions will result in vague responses. And while respondents may try to answer honestly, they can’t because they don’t get the question.
- Design the questionnaire instrument so that respondents know where you are going with the questions. Question order and flow, along with a clear purpose statement and a good set of instructions, go a long way in gaining respondent confidence that the use of the data is for good—rather than the bad.
- Introduce the questionnaire early in the program. This is especially true when you plan to follow-up on the application, impact, and ROI of a program. Walk them through the questions so they know what to expect and how you plan to use results from each question.
- When introducing the questionnaire, remind participants you are not collecting data to evaluate their performance. Rather, you are collecting data to evaluate the program’s performance. Evaluation is about the program, not the people.
After analyzing the data, share the results with respondents. Tell them what the results mean to the program, and how you plan (and do) use them.
