LLM Evals · Turn quality into a signal
How do you keep judge cost under control?
Short answer
Sample rather than judge everything, use deterministic checks where possible, cache verdicts, batch calls and reserve expensive judges for high-risk cases.
Judging every output with a large model can cost more than the feature itself. The goal is not maximum coverage per release; it is enough signal to make a decision. A tiered approach usually wins: cheap checks first, judges on a sample, humans on escalations.
- Use schema checks, unit tests and exact rules before any model call.
- Judge a stratified sample, not the full eval set, when the signal is stable.
- Cache verdicts by input, output and rubric hash.
- Use a smaller judge for routine categories and a stronger one for ambiguous cases.
- Budget judges per release and track cost per eval run.
Follow-ups you should expect
Interviewers use these to check whether the first answer was understanding or memorisation.
- 1How would you choose the sample size for a judge run?
- 2When would caching a verdict be invalid?
- 3How would you compare the cost of a judge with human review?
Say it in the interview
I do not judge everything. I run deterministic checks first, then judge a stratified sample with a cheaper model, and reserve a stronger judge or humans for ambiguous and high-risk cases. Verdicts are cached by input, output and rubric hash. I also track cost per eval run. The point is enough signal to make a release decision, not perfect coverage of every output.
Use this as a model, not a script. Saying it in your own words is what makes it survive follow-up pressure.
Related questions
Reading is step one
Now say it out loud.
Understanding an answer and delivering it under pressure are different skills. Rehearse this topic in a live LLM Evals mock interview and get scored feedback on depth, clarity and confidence.