With the rise of sites like SurveyMonkey, surveys have become so ubiquitous, so accessible, so easy — anyone can do them. It’s truly the democratization of survey research. But just because anyone can do a survey, doesn’t mean they should.
Doing a survey is one thing. Doing a survey right is an entirely different proposition. Responsible survey researchers will take time to learn the fundamentals of the methodology, before jumping too fast into building a survey and deploying it willy nilly without thinking through every aspect from instrument design to recruitment strategy. If you do it right, planning the survey should take about 90% of the time.
Programming a survey into modern software like SurveyMonkey, Qualtrics or Typeform is really the least of your efforts when doing a good and credible survey.
Survey research comes up a variety of UX research contexts — the most common is probably when assessing UX metrics, such as the 10 questions in a SUS questionnaire. In the case of SUS, NPS and other well-established instruments, we don’t need to worry about writing the questions — they are predetermined. But, depending on the context, we may need to worry about things like sample size and ensuring the results are not just statistical noise. For exploratory UX research or homegrown metrics, we may also find ourselves designing questions to learn about user needs and experiences.
What does doing good, credible, responsible survey research even mean, anyway? Generally it means taking all the steps that will allow you to obtain statistically significant, unbiased findings that can be extrapolated to your full population of interest. Without doing this, you’ll at best get useless information. At worst, you’ll get results that could seriously mislead your team.
Add to that the challenges of even getting people to complete your survey in the first place, and you’ve got a lot more to contend with than you might have first realized when you created that SurveyMonkey account.
So, here’s the Good, the Bad, and the Ugly things to watch out for in survey research (along with some tips for survey success)
THEY’RE QUICK TO COMPLETE - Since surveys don’t require too much of people’s time to complete, you’ve got a decent chance at getting a good number of respondents. However, this falls apart quickly, depending on the length of your survey. The company Survicate did a study where they analyzed nearly 270,000 resposes from surveys conducted by their customers and found that completion rates go from 83% with up to 3 questions all the way down to 42% for 15 questions. Keep those surveys short and snappy, people!
Paying people to complete your survey helps. For the general population $2 per survey is a common incentive. For specialized or high-paid professions, where time is particularly valuable (such as health care professionals) or cases where you need a high percentage of recipients to complete it, you’ll need a much heftier incentive. One study of fair market value for health care professional survey incentives found that incentives need to be $30-50 for a 15 minute survey in order to get adequate responses from physicians.
Check applicable laws and regulations when doing “drawings” for incentives such as gift cards, so you don’t run afoul of the laws governing sweepstakes and lotteries.
YOU GET STRUCTURED DATA - The data you get is highly structured, typically. This makes it a lot quicker to analyze than sifting through open-ended interview responses.
THE BAD & THE UGLY:
IT’S HARD TO WRITE GOOD QUESTIONS - There’s actually a real art to writing good questions that avoid bias and are clear to respondents. People who are deploying surveys in the name of generating credible research results absolutely must master the art of good questions to avoid introducing bias and other errors into their results. Simply put, the answers and insights you will gain from doing a survey are only as good as the questions you ask.
What to do? (1) Learn how to write good questions. In my opinion, “The Complete Guide to Writing Questionnaires” is the best and most comprehensive resource for the fine art of writing good questions. Everyone deploying surveys should read this. (2) Test your survey questions. Yes, test them. With actual people. Test them with colleagues, and ideally test with actual users to make sure your target population can understand what you’re asking. This is called cognitive testing of your survey. It’s also called QA. Do it.
ERRORS DUE TO WRONG SAMPLE - You need to put some real thought and effort into getting an appropriate sample to complete your survey. Ideally, your sample would represent a truly random sample from your population of interest — offering a higher incentive can help with this.
In practice, convenience samples are also used — which is when respondents are selected simply based on their accessibility—as when an online intercept or customer email list is used. This is most valid when the population of interest is defined as “users of this website” or “people on this email list”.
Sample Size also matters. A lot. It will determine whether your results are statistically significant or not. Without the right sample size, you run the risk of reporting “findings” that are actually due to random noise.
To properly calculate sample size, you need to know how precise the results need to be — what’s the desired margin of error, how variable the data is likely to be (or a worst case estimate if no other data is available), the critical difference (smallest difference you need to be able to detect) and the level of statistical confidence desired. Jeff Sauro has written some of the best descriptions about how to do all this for applied UX research that I’ve ever seen — including for discrete and continuous survey questions. Check his blog or his book, Quantifying the User Experience.
Top Tips for Survey Success:
Start your survey research with a well thought out research question and plan. Know what decisions will be made or actions taken by stakeholders based on the results, if any. Think about how you’ll analyze the data.
Consider the “Good” “Bad” and “Ugly” factors described above in designing your survey research and take your time making sure that your survey will be credible and provide the needed answers at the needed level of precision and confidence.
Craft a fantastic recruitment email or intercept. You need to really ‘pitch’ your survey — convincing potential respondents that you are trustworthy and the research is valuable and won’t take too much of their time.
Use a survey platform that makes taking your survey actually fun. Many platforms have interactive, visual or other types of questions that can take your survey beyond the simple a,b,c’s of multiple choice questions.
Consider hiring an expert. If the results are particularly high stakes, or if major financial decisions will be made based on the results, as is often the case with market research, you’d be wise to hire a research professional who has experience in all aspects of survey research, from design and recruitment to data collection and statistical analysis.
Of course, not all survey research is so high stakes, so don’t shy away from doing a survey when it’s appropriate for the information you seek to gather — just make sure your survey is the best it can be by following the well-established practices of credible survey research.