LLM Evals · Turn quality into a signal
What biases do LLM judges have?
Short answer
Judges favour longer answers, their own style, earlier or later positions in pairwise comparisons, confident tone and familiar formats. Randomise, constrain and calibrate to reduce these effects.
A judge is a model, so it has preferences. Verbosity bias rewards length even when detail is irrelevant. Position bias favours the first or second candidate. Self-preference favours outputs from the same model family. Format bias rewards bullet points regardless of content.
- Randomise candidate order and average both directions.
- Control for length by trimming, stratifying or explicitly penalising verbosity.
- Use a different model family for judging when feasible.
- Give the rubric anchored examples of each score level.
- Audit disagreements stratified by output length and format.
Follow-ups you should expect
Interviewers use these to check whether the first answer was understanding or memorisation.
- 1How would you test for position bias?
- 2Why is self-preference a problem for model comparison?
- 3How do you handle a judge that always gives middling scores?
Say it in the interview
Judges tend to favour longer answers, confident tone, familiar formats and their own model family. Pairwise judges can also favour whichever answer they see first. I randomise order, average both directions, control for length and use a different model family when I can. I also audit disagreements, because a judge that never disagrees with itself may simply be insensitive rather than accurate.
Use this as a model, not a script. Saying it in your own words is what makes it survive follow-up pressure.
Related questions
Reading is step one
Now say it out loud.
Understanding an answer and delivering it under pressure are different skills. Rehearse this topic in a live LLM Evals mock interview and get scored feedback on depth, clarity and confidence.