LLM Evals · Turn quality into a signal
How do you write labeling guidelines?
Short answer
Define each label with positive and negative examples, specify edge-case rules and tie decisions to user impact. Test the guidelines on a sample and revise where reviewers disagree.
Ambiguous labels produce noisy evals, and noisy evals lead engineers to optimise the wrong thing. Guidelines should be specific enough that two trained reviewers reach the same decision on most cases, with a documented process for genuine ambiguity.
- Define the decision, not the general topic: 'supported by the cited passage' rather than 'good answer'.
- Include borderline examples and the reasoning behind each call.
- State what to do with partial answers, refusals and multiple valid answers.
- Run a calibration batch and measure agreement before scaling labelling.
Follow-ups you should expect
Interviewers use these to check whether the first answer was understanding or memorisation.
- 1What agreement level would make you comfortable?
- 2How would you handle a label that requires domain expertise?
- 3How do guidelines differ for binary versus graded labels?
Say it in the interview
Good guidelines define the decision, not the topic. For groundedness, for example, the rule is whether each claim is supported by the cited passage, with examples of partial support and unsupported but true claims. I include borderline cases and a process for ambiguity, then run a calibration batch and measure agreement. If reviewers disagree, the guideline is usually the problem, not the reviewers.
Use this as a model, not a script. Saying it in your own words is what makes it survive follow-up pressure.
Related questions
Reading is step one
Now say it out loud.
Understanding an answer and delivering it under pressure are different skills. Rehearse this topic in a live LLM Evals mock interview and get scored feedback on depth, clarity and confidence.