Just like a good survey can help a company achieve its goals or resolve its issues, a poorly designed survey can be misinterpreted or misapplied and adversely affect a company’s performance. In order to make good data driven decisions you have to ensure that the data you are basing your decisions on is both valid and reliable. Here are 6 common survey mistakes:
Skipping the Introduction
Make sure you tell the respondents what this survey is about and why they should complete it. A valid rationale means a more committed survey-taker and avoidance of one of six common survey mistakes.
Sampling Bias
Is your sample representative of the population you are interested in? The demise of landlines has made getting a fair, target population difficult. Easy to reach people are not necessarily representative of the population you are testing. Monitor your sample demographics throughout the data collection process.
Poor Survey Design
DIY survey software has been a boon to getting more data easily but there is a science, with extensive academic research supporting it, to writing an effective, fair survey. Poor scaling. A four option scale can give wildly different results than a five option scale. Biased framing. A classic framing example is that more people will rate ground beef better if it’s framed as 80% lean versus 20% fat. Poor surveys use framing to create push polls that yield the desired results. Check the question wording before accepting the results. Social desirability bias. People will say they watch documentaries when they spend all their time watching reality TV. Mitigate this type of bias by using careful wording that reduces the perceived risk of choosing undesirable responses and lessens the pressure to select socially desirable answers. Review surveys holistically and see if they are internally consistent. Only 10% of respondents say they would purchase an electric vehicle but 30% say they would purchase a Tesla, which only makes electric cars. Something is wrong and you’ve just made another one or two or three of the common survey mistakes.
Poor Questions
The most important aspect of your survey, the questions, is also a minefield of possible problems. Flawed questions can be written above or below the knowledge level of the sample, be leading or biased, touch on more than one issue but only allow one answer, have response choices that are not mutually exclusive, have too many with open-ended answers that are difficult to quantify fairly and, finally, only offering a binary choice when more nuanced choices would provide more accuracy. An example of this last flawed question mistake is the Brexit referendum. The British electorate was only given the binary choice of stay or leave. 52% said leave, but new research from YouGov suggests that only 33% of voters preferred a hard leave option, a very consequential common survey mistake for the EU and Britain.
Not Pretesting
Bypassing the final step of testing could lead to disaster. A small group of individuals testing your survey can bring obvious errors to light, help you make the survey easier to take, insure the logic of the survey is good and eliminate any obvious biases. A simple step to solve many common survey mistakes.
Ready For Public Consumption?
Presentation of the results is critical to the success of the survey. Are you providing feedback that can be used and subsequently assessed for impact? Are you drawing a correlation between the money spent on the research and the business results obtained? Are your methodological approaches and questions easily accessible for credibility review?
There are many ways to go wrong in designing, conducting and summarizing a market research survey and committing any of these common survey mistakes. Interested in participating in this exciting industry? Contact Smith Hanley Associates’ Market Research Executive Recruiter, Lindsey Bartlett at lbartlett@smithhanley.com.
For a good article on this topic visit Quirks.com.
