LLM Evals · Turn quality into a signal
How do you pick metrics for a specific task?
Short answer
Start from the decision the feature supports and the failure that would hurt most. Choose observable criteria that distinguish good from bad, then validate the metric against human judgement.
A metric should answer a question the team will act on. For extraction, field accuracy and schema validity matter. For a support assistant, groundedness and correct escalation matter. For a creative tool, novelty and adherence to constraints matter more than exact wording.
- Name the user harm and the business cost of each failure mode.
- Prefer observable, checkable criteria over vague quality scores.
- Use multiple metrics when trade-offs exist, and report them separately.
- Validate the metric on labelled examples before optimising against it.
Follow-ups you should expect
Interviewers use these to check whether the first answer was understanding or memorisation.
- 1What would you measure for a summarisation feature?
- 2How do you handle a metric that improves while user satisfaction falls?
- 3When is a human preference score better than an automatic metric?
Say it in the interview
I start from the failure that would hurt the user most. For extraction that is field accuracy and schema validity; for a support assistant it is groundedness and knowing when to escalate. Then I pick observable criteria and check them against human labels. I keep multiple metrics separate when there are trade-offs, because a single score hides the decision. If a metric cannot be computed reliably, it is not useful.
Use this as a model, not a script. Saying it in your own words is what makes it survive follow-up pressure.
Related questions
Reading is step one
Now say it out loud.
Understanding an answer and delivering it under pressure are different skills. Rehearse this topic in a live LLM Evals mock interview and get scored feedback on depth, clarity and confidence.