LLM Evals · Turn quality into a signal
What is eval-driven development?
Short answer
Write the evaluation before or alongside the feature, define what good looks like, then iterate against that signal. It is test-driven development adapted to probabilistic behaviour.
In eval-driven development the quality bar exists before the prompt is tuned. You collect representative inputs, define expected behaviour, and build a scorer. Then every change is measured against the same bar rather than judged by a few cherry-picked examples.
- Start with a small set of real cases and explicit expectations.
- Build the simplest scorer that distinguishes good from bad.
- Iterate on prompt, retrieval or model, and re-run the suite.
- Promote production failures into permanent eval cases.
Follow-ups you should expect
Interviewers use these to check whether the first answer was understanding or memorisation.
- 1How would you start if the feature is still a prototype?
- 2What is the risk of overfitting to a small eval set?
- 3How does this change the review process for prompt changes?
Say it in the interview
Eval-driven development means defining quality before tuning the model. I gather real inputs, write down expected behaviour, and build a scorer, then every prompt or retrieval change runs against the same set. It is not about perfect metrics; it is about having a consistent signal instead of judging two cherry-picked examples. Production failures become new cases, so the suite grows with the product.
Use this as a model, not a script. Saying it in your own words is what makes it survive follow-up pressure.
Related questions
Reading is step one
Now say it out loud.
Understanding an answer and delivering it under pressure are different skills. Rehearse this topic in a live LLM Evals mock interview and get scored feedback on depth, clarity and confidence.