LLM Evals · Turn quality into a signal
Pairwise vs pointwise judging?
Short answer
Pointwise scores each output against a rubric; pairwise compares two outputs directly. Pairwise is better for subtle preferences, pointwise is better for absolute quality and large-scale monitoring.
Pointwise judging asks how good this answer is. It is cheap to aggregate and easy to track over time, but absolute scales drift between prompts and models. Pairwise judging asks which of two answers is better, which is more sensitive to small differences but does not produce an absolute score and needs order randomisation.
- Use pointwise for dashboards, thresholds and regression gates.
- Use pairwise for prompt A/B decisions and subtle quality differences.
- Pairwise results do not tell you whether both answers are bad.
- Combine them: pairwise to choose, pointwise to monitor.
Follow-ups you should expect
Interviewers use these to check whether the first answer was understanding or memorisation.
- 1Which would you use to decide between two prompts?
- 2How do you detect that both candidates are poor?
- 3Why can pointwise scores drift over time?
Say it in the interview
Pointwise gives an absolute quality score, which is good for dashboards and gates. Pairwise asks which of two answers is better, which is more sensitive to small differences and useful for choosing between prompts. Pairwise does not tell me whether both are bad, so I combine them: pairwise to make a decision, pointwise to monitor over time. Pairwise also needs order randomisation.
Use this as a model, not a script. Saying it in your own words is what makes it survive follow-up pressure.
Related questions
Reading is step one
Now say it out loud.
Understanding an answer and delivering it under pressure are different skills. Rehearse this topic in a live LLM Evals mock interview and get scored feedback on depth, clarity and confidence.