Tips to Protect the Quality of Survey Data
When collecting survey data, businesses want to ensure survey responses are of high quality and are reliable.
However, this is not an easy task as many different factors can compromise the validity and integrity of survey results. It is especially difficult to acquire high quality data when collecting data through online survey tools as it is harder to discern respondent identities virtually.
If researchers fail to filter out unreliable, unqualified, or fraudulent responses, the data produced may be skewed and can invalidate the findings of the study.
Within this article, we will explore the different ways in which you can protect the integrity of your survey data to ensure the collection of reliable and useful insights.
5 Tips to Ensure the Integrity of Survey your Survey Data
Let’s investigate the different measures you can take to maintain the integrity of your survey data to obtain high-quality information:
- Implementing Multiple Trap Questions
- Incorporate Human Monitoring
- Check IP Addresses
- Check for Speeders
- Use a Red Herring Question
Implementing Multiple Trap Questions
Trap questions refer to questions that are included in a survey to test whether respondents are answering the survey carefully and honestly. Inattentive respondents introduce noise into your data sets and can weaken the correlations discovered between variables, increasing the likelihood of null findings.
Trap questions are meant to be easy to answer, sometimes even obvious, and are meant to test survey respondents’ honesty and attentiveness rather than their knowledge. They are useful to all kinds of surveys as even high-cost specialized panels can include inattentive responses.
Thus when designing a survey using a survey software, it is recommended to include more than one trap question as multiple trap questions can help weed out a majority of the false responses while just one trap question may not do so.
Incorporate Human Monitoring
Human monitoring can be incorporated into a research design in many ways. For example, researchers can review the open-ended answers in the survey before removing certain respondents. Humans will be able to discern odd responses better than technology as this requires a level of subjectivity that technology may lack.
This method is especially useful in identifying survey bots. Survey bots refer to scripts, or “bots”, that can automatically fill in survey questions whether they take the form of question bubbles or even open text spaces.
Survey bots are a large concern for researchers as they can seriously compromise the validity of a study. Human monitoring can help minimize this bias as it will help identify unusual or low-effort responses, especially in open-ended questions.
Check IP Addresses
Setting advanced cookies or tracking respondents’ IP addresses will attach each response to a physical address, allowing you to discern if multiple responses have been submitted from a single IP address.
This measure is very effective in identifying imposters. Imposters are those respondents that provide false demographic data to qualify for the survey’s target population. Even if just a small percentage of survey responses are imposters, they can really skew the findings of the study, producing invalid results.
This can compromise data in market research surveys, leading to incorrect findings using market research tools. By tracking IP addresses and using cookies, you can minimize this risk as it will allow you to eliminate respondents that misrepresent themselves.
Check for Speeders
Speeders are respondents who work through surveys quickly without paying much mind to each question individually. As these respondents aren’t devoting their attention to reading and understanding survey questions, their responses may not reflect their true answers.
A great way to check for speeders is to eliminate responses that were completed in a really short period of time. Most survey platforms provide data reflecting the log time each respondent took to complete the survey.
This reflects the time the respondents took to complete the survey from start to finish. If you know that your survey takes an average of 8 minutes to complete and you have multiple responses from respondents that took just 1 or 2 minutes, this can be a red flag. Such responses should be reviewed and their validity should be evaluated. As per evaluation, if any responses seem lax or implausible, they can be eliminated.
Use a Red Herring Question
A red herring question is a question that is included in surveys as a quality control measure. It involves placing an oddball question amid the series of regular survey questions to identify which respondents are properly engaged with the survey and which did not.
A red herring question is a type of trap question that allows researchers to skim survey data for invalid or fake responses.
An example of a red herring question would be to simply ask respondents to select a certain number on a 0 to 10 rating scale. Respondents who don’t select the number they were directed to, and hence failed to complete this basic task, can be eliminated from the study as they are likely not reading the questions being asked while responding.
This method can help eliminate speeders, as well as ‘straightliners’. Straightliners are respondents who select the same answer for nearly every question in the survey with the sole purpose of submitting their response rather than responding honestly and objectively.