LLM Evals · Turn quality into a signal
How do you calibrate a judge against humans?
Short answer
Have humans label a representative sample, compare judge verdicts, inspect disagreements and adjust the rubric or judge until agreement is acceptable. Re-calibrate whenever the judge or task changes.
Calibration answers whether the judge measures what humans care about. A judge can be internally consistent and still systematically wrong. The comparison should use the same cases, the same rubric and blinded human labels.
- Select a stratified sample that includes edge cases, not only typical outputs.
- Collect independent human labels before showing judge results.
- Report agreement overall and by category, length and output type.
- Read disagreements to distinguish rubric ambiguity from judge error.
- Re-calibrate after model, prompt or rubric changes.
Follow-ups you should expect
Interviewers use these to check whether the first answer was understanding or memorisation.
- 1What agreement level would justify using the judge at scale?
- 2How would you handle a category where humans disagree with each other?
- 3How often would you re-calibrate?
Say it in the interview
I have humans label a representative sample, including hard cases, before I show them the judge's results. Then I compare overall and by category, and read the disagreements to see whether the rubric is ambiguous or the judge is wrong. I keep a held-out calibration set so I do not tune the judge to the same examples. Whenever the judge model or rubric changes, I re-calibrate.
Use this as a model, not a script. Saying it in your own words is what makes it survive follow-up pressure.
Related questions
Reading is step one
Now say it out loud.
Understanding an answer and delivering it under pressure are different skills. Rehearse this topic in a live LLM Evals mock interview and get scored feedback on depth, clarity and confidence.