Decreasing respondent friction to increase data quality
Introducing Convo: effortless data transfer that cut survey length and multiplied response richness.
Situation
In market research, there is a catch-22: the most actionable data comes from in-depth interviews, but the rigour of quantitative data requires scale. The cost of IDIs makes collecting at scale impractical. Many research firms claim to deliver “the depth of qualitative data at the speed and scale of quantitative,” but online surveys are cheap for a reason.
Anyone who has worked with survey data quickly realises that respondents fatigue faster than a toddler and look to skip through as fast as possible, to the detriment of data quality. From a respondent’s perspective, the faster they complete surveys, the more they can earn. Responses are bare, stopping the moment they hit the minimum word count.
Task
To truly achieve rich, actionable data with the rigour of scale, we targeted respondent fatigue with an engaging experience that promotes an effortless transfer of information. Removing these barriers is the key mindset that most market research firms brush over. After all, what could be easier than typing?
Goal and KPIs:
- Reduce respondent fatigue (driven primarily by LOI)
- Increase the richness and quality of data (human-evaluated quality, response length)
Action
Exploratory phase: Our researchers surfaced the problem clearly. Years of accumulated open-ended data told the same story: respondents dislike open-ended questions. They require time that could be spent earning a reward on another survey.
Hypothesis: Respondents can read, think, and speak much faster than they type. Talking to a survey should be faster than writing a paragraph.
Secondary research: Extensive secondary research into voice input for online surveys suggested a historic preference for typing, primarily due to technical limitations and habit. However, as voice messaging has become ubiquitous in modern communication, we anticipated a shift in respondent sentiment. Existing studies were dated and didn’t align with our specific use case, so we proceeded with primary research to validate.
People don’t know what they want until you show it to them.
Steve Jobs
Prototypes: We evaluated interface styles including voice memos, live voice-to-voice conversations, live transcription, and more. After A/B testing showed promising results for one variant (−17% LOI, +32% response length, +18% data quality), we developed it into Convo and monitored KPIs through live client projects, seeing long-term, sustained improvement.
Result
LOI: −31%
Response length: +42%
Data quality: +8%
Below is an example of typed verbatims compared to spoken verbatims from the same prompt: Tell me a story and describe your emotions.
*Data changed to protect client and respondent agreements.
Before
"Calm and tired, wanting to be productive"
"Full and happy when I eat lunch"
After
"I was with my husband and my three children. We were sitting around in the kitchen talking about the great day we had. We felt happy, we were having fun. We decided to play a game even. This time we had together was really special to me."
From a client perspective, this created more actionable insights:
We’ve never got anything this good before!
T5 US food and beverage company
It got us to a deeper level, it gave us more to think about!
T5 worldwide energy provider
While the initial rollout was a success overall, technical difficulties for edge-case users caused some drop rates to increase more than expected. With Protobrand’s fail-fast culture, this was expected. Later iterations continued to reduce drop rate and improve the respondent experience.