LLM Evals · Turn quality into a signal
How do you design a scoring rubric?
Short answer
Define observable criteria with anchored levels, separate independent dimensions, and make each level distinguishable with examples. Avoid vague labels like 'excellent' without evidence.
A rubric turns a quality judgement into a repeatable decision. It should name what to inspect, how to score it and what distinguishes adjacent levels. Combining several qualities into one number makes disagreement likely and debugging impossible.
- Score independent dimensions separately: faithfulness, completeness, format.
- Anchor each level with a concrete example or test.
- Use a small number of levels, typically three to five.
- State how to handle partial credit and missing evidence.
- Pilot the rubric and measure agreement before scaling.
Follow-ups you should expect
Interviewers use these to check whether the first answer was understanding or memorisation.
- 1How many levels should a rubric have?
- 2What would you do about a criterion with poor agreement?
- 3Why separate faithfulness from completeness?
Say it in the interview
I design rubrics around observable criteria and score independent dimensions separately, because faithfulness and completeness can fail in different ways. Each level needs a concrete anchor, not an adjective like excellent. I keep the number of levels small and test the rubric on a pilot batch to see where reviewers disagree. If a criterion cannot be checked from the output and evidence, it does not belong in the rubric.
Use this as a model, not a script. Saying it in your own words is what makes it survive follow-up pressure.
Related questions
Reading is step one
Now say it out loud.
Understanding an answer and delivering it under pressure are different skills. Rehearse this topic in a live LLM Evals mock interview and get scored feedback on depth, clarity and confidence.