LLM Evals · Turn quality into a signal
How does LLM-as-judge work?
Short answer
A model grades an output against a rubric, returning a score or verdict. It scales subjective evaluation, but needs a narrow rubric, reference evidence and calibration against human labels.
The judge receives the task, the output and often a reference or source evidence, then applies criteria. It can score pointwise, compare candidates pairwise or extract structured findings. The quality of the verdict depends more on the rubric and evidence than on the judge model's general intelligence.
- Use explicit, observable criteria and a fixed output schema.
- Provide the source evidence for groundedness judgments.
- Randomise order for pairwise comparisons.
- Measure judge-human agreement before scaling.
- Pin the judge version and re-validate when it changes.
Follow-ups you should expect
Interviewers use these to check whether the first answer was understanding or memorisation.
- 1When would you prefer a deterministic check to a judge?
- 2How would you write a judge prompt for citation support?
- 3What would you do if the judge disagrees with your reviewers?
Say it in the interview
An LLM judge applies a rubric to an output and returns a verdict. It lets me grade subjective qualities at scale, but it is only as good as the rubric and the evidence I give it. I use deterministic checks where they exist, such as schema validation or code execution. For the rest, I validate the judge against human labels and pin its version, because a silent judge change would shift all my scores.
Use this as a model, not a script. Saying it in your own words is what makes it survive follow-up pressure.
Related questions
Reading is step one
Now say it out loud.
Understanding an answer and delivering it under pressure are different skills. Rehearse this topic in a live LLM Evals mock interview and get scored feedback on depth, clarity and confidence.