LLM Evals · Turn quality into a signal
How do you turn user feedback into evals?
Short answer
Capture explicit and implicit signals, triage them into failure categories, reproduce the trace, then add a labelled case to the golden set. Do not treat a thumbs-down as a complete label.
Feedback is a detection mechanism, not a dataset. A thumbs-down says something went wrong, not what or why. To become an eval case, it needs the trace, a human or evidence-based label, and a reproducible input.
- Collect explicit ratings and implicit signals such as retries, edits and escalations.
- Group feedback by failure category and frequency.
- Pull the trace to see whether retrieval, generation or policy failed.
- Write an expected behaviour and add the case to the versioned set.
- Close the loop by tracking whether the fix holds.
Follow-ups you should expect
Interviewers use these to check whether the first answer was understanding or memorisation.
- 1How would you prioritise which feedback to turn into tests?
- 2Why is a thumbs-down not a sufficient label?
- 3How do you avoid overfitting to the loudest users?
Say it in the interview
Feedback tells me something went wrong, but not what. I triage it by failure category, pull the trace to see whether retrieval, generation or policy caused it, and write an expected behaviour before adding it to the golden set. I also pair explicit ratings with implicit signals and sampled review, because only motivated users leave feedback. Then I track whether the fix actually holds.
Use this as a model, not a script. Saying it in your own words is what makes it survive follow-up pressure.
Related questions
Reading is step one
Now say it out loud.
Understanding an answer and delivering it under pressure are different skills. Rehearse this topic in a live LLM Evals mock interview and get scored feedback on depth, clarity and confidence.