A previous research agency client once ran an online quantitative study where nearly three in four respondents produced clearly contradictory responses to two agreement scale statements. The two statements were polar opposites - deliberately designed to help identify “low quality” respondents - and we found 41% agreed with both statements, while 31% disagreed with both. This left just 28% producing responses which could be described as consistent…!

Given these findings, how can end users of online quantitative research have confidence in how the average respondent answers ANY survey question?

A typical solution is to remove “low quality” respondents – however, if numbers are large (as above) this is unlikely to be desirable. Further, unlike in the clear-cut example above, this exercise can sometimes be highly subjective as well as time-consuming.

Instead, confidence in online quantitative research findings can be significantly enhanced through the three tips below:

1. LIMIT USAGE OF GRIDS: First, research has found that when compared to individual statements, grids produce greater variability in response (1). This is consistent with the idea that respondents are more likely to produce random, haphazard responses when answering in grid format.

2. KEEP SURVEYS SHORT: Second, confidence can be increased through not asking too much of respondents. One way to do this is to ensure quantitative surveys are no longer than they really need to be. One recent study suggested the ideal length for an online survey is a median of 10 minutes – and the maximum survey length should be 20 minutes (2).

3. VALIDATE USING OTHER SOURCES: Finally, it is often valuable, wherever possible, to validate key data points against other, non-survey sources. These could include for example qualitative research findings, expert opinions, transactional (online or otherwise) data, and social media data. The greater the range of non-survey sources telling a similar story - the more confidence we can have in our own findings.

Of course, in terms of individual questions, we can never completely eradicate the chances of a question being biased in some way – and in this respect the complete “truth” will always remain elusive. However, the above offers three simple and practical ways to ensure significantly greater confidence in online quantitative research findings.

Chris Harvey
Founder, Activate Research

References
1. https://www.surveymonkey.co.uk/curiosity/putting-matrix-grids-to-the-test/
2. Revilla, M., & Ochoa, C. (2017). Ideal and Maximum Length for a Web Survey. International Journal of Market Research, 59, 557-565.