Survey design is a balance between art and science. Knowing the types of error that affect the quality of a survey and the subsequent results is an important first step in addressing the science side of the scale.
According to Don Dillman and his colleagues (Internet, Mail, and Mixed-Mode Surveys: The Tailor-Made Design Method, Sage Publications, 2008), there are four types of errors we need to consider when designing surveys:
- coverage error
- sampling error
- response error
- measurement error.
Coverage error refers to the opportunity we give all potential respondents to respond to the survey. For example, if you want to survey employees to determine their overall perception of the learning function, and 20 percent of the employees work in the field without access to a computer, would you administer your survey using an electronic channel of delivery?
Many would argue Yes! And then ignore those people in the field. But if the opinions of your field people are just as important as those employees with electronic access, you need an alternative approach to delivering the survey to the field. Have you thought of using the internal mail courier, or how about postal service? Recently, we ran across a group that actually faxes surveys to respondents.
Sampling error occurs when the sample is too small to adequately infer survey results to non-respondents. There is a lot of confusion about sampling and when and how to use it. Sampling was devised as a way to collect data from the few to infer to all. But “all” depends on the information you seek.
Your population is defined as the group of people who have the information you want. So, if you want to survey the general population of 50,000 employees about their opinion toward the new smoking ban, then sampling is a way to avoid costs of collecting data from the entire population. A simple tool to help you calculate sample size is located at http://www.raosoft.com/samplesize.html.
But if you want to determine the business impact and ROI of a leadership program for the 25 people who completed the program, your population is the 25 and you administer the survey to the entire population of 25. Your results reflect the results of the 25, but do not suggest the same results for anyone outside that group.
Response rate error is sometimes confused with sampling. Whereas sampling concerns gathering data from a small, representative group from the population, response rate correlates to getting the data back from those to whom you’ve administered the survey. In order to ensure a positive response rate, we suggest you develop a survey administration plan that lists steps you will take before administering the survey, during the data collection period, and after the survey closes that will entice people to respond.
Measurement error is a critical type of error on which we should place laser-like focus. Unfortunately, it is an area on which less attention is paid than response rate. And like it or not, if you have 100 percent response rate on a survey with poorly designed questions, you have nothing. So, measurement error is the first place to focus when designing surveys.
To mitigate measurement error, make sure you ask the right questions, the right way. How do you know you are asking the right questions? Take a look at your objectives. How do you ask the questions the right way? Write questions that allow respondents the opportunity to provide the most accurate responses possible.
