LLM Evals · Turn quality into a signal
How do you measure agreement between labelers?
Short answer
Use percent agreement for a quick check and Cohen's kappa or Krippendorff's alpha to account for chance. Low agreement means the guideline or task is ambiguous, not that the metric is broken.
Agreement estimates whether a label is reliable enough to use as ground truth. Percent agreement is intuitive but inflated when one label dominates. Chance-corrected statistics such as kappa or alpha are more honest, especially for imbalanced classes.
- Have multiple reviewers label the same overlapping sample.
- Report both raw agreement and a chance-corrected statistic.
- Inspect disagreement cases rather than only the score.
- Tie the acceptable threshold to the cost of a wrong label.
Follow-ups you should expect
Interviewers use these to check whether the first answer was understanding or memorisation.
- 1Why can percent agreement be misleading?
- 2What would you do about a category with very low agreement?
- 3How many overlapping cases do you need?
Say it in the interview
I have reviewers label an overlapping sample and report both raw agreement and a chance-corrected measure like kappa. Percent agreement can look good simply because one label is common. When agreement is low, I read the disagreements, because the problem is usually an ambiguous guideline or a task that needs a clearer definition. High agreement is not proof of correctness either, so I calibrate against expert calls.
Use this as a model, not a script. Saying it in your own words is what makes it survive follow-up pressure.
Related questions
Reading is step one
Now say it out loud.
Understanding an answer and delivering it under pressure are different skills. Rehearse this topic in a live LLM Evals mock interview and get scored feedback on depth, clarity and confidence.