LLM Evals · Turn quality into a signal
How do you know an improvement is real?
Short answer
Compare paired results on the same cases, estimate uncertainty with confidence intervals or bootstrap tests, and account for multiple comparisons. A small gain on a small set is often noise.
An eval score is an estimate from a sample. If the set is small or the metric is noisy, a two-point improvement may be indistinguishable from random variation. Reporting a single number without uncertainty encourages overconfident decisions.
- Use paired comparisons when the same cases are evaluated by both versions.
- Bootstrap confidence intervals for aggregate metrics.
- Check per-category results; an overall win can hide a serious regression.
- Correct for multiple comparisons when testing many categories.
- Prefer larger, more representative sets over repeated peeking.
Follow-ups you should expect
Interviewers use these to check whether the first answer was understanding or memorisation.
- 1Why are paired comparisons more powerful than independent samples?
- 2How would you handle a metric that is very noisy?
- 3What would make you delay a release despite a higher average?
Say it in the interview
I treat an eval score as an estimate, so I want uncertainty around it. I use paired comparisons on the same cases and bootstrap confidence intervals, and I check categories because an overall win can hide a regression. I also adjust for testing many subgroups. A statistically real improvement still has to be worth the latency or complexity, so significance is not the same as shipping.
Use this as a model, not a script. Saying it in your own words is what makes it survive follow-up pressure.
Related questions
Reading is step one
Now say it out loud.
Understanding an answer and delivering it under pressure are different skills. Rehearse this topic in a live LLM Evals mock interview and get scored feedback on depth, clarity and confidence.