Optimising model performance via human feedback
Automate time-intensive workflows to allow for shorter timelines and lower asking prices.
Situation
Since the fall of 2022, clients have been asking with more vigour for faster, cheaper, and better research. In the past, this violated the sacred triple constraint rule (you can only have 2/3 of fast, cheap, and good). The rise of LLMs after GPT 3.5 in November 2022 has kicked off a global race across all industries to nail all three at once, and market research is no different.
Task
Reducing labour costs per project and increasing research speed are the primary ways to achieve all three at once. So, the task became: automate time-intensive researcher workflows to decrease the overall timeline and cost of a project, without compromising quality.
Action
We identified the primary target to be thematic coding, a judgement-heavy, nuanced task of inductively logging ideas across hundreds or thousands of respondent verbatims. Our researchers were spending an average of 8 hours per project coding open ended data, a time-intensive (and thus cost-intensive!!) bottleneck for short timeline projects.
After evaluating off-the-shelf products (Canvs AI, Codeit, Ascribe, Caplena) and finding them lacking in cost or fit, we built our own solution. The current working product chains multiple LLM calls together, cycling through revisions, validations, and human-in-the-loop steps to produce an accurate and actionable codebook.
KPIs:
- Time saved per project
- Performance benchmarked against human-led coding: semantic similarity of themes (LLM-as-judge) and salience difference for code matches.
Result
This project is still in development, so final KPIs have not been recorded. Early testing is promising though:
- Time saved per project: 7.5 hours, our target is 30 minutes down from 8 hours (at a cost of <$10 in API calls)
- Performance benchmarked against human-led coding: average salience difference of only 2.13%