LLM Evals · Turn quality into a signal
How do you measure answer relevance?
Short answer
Check whether the answer addresses the question asked, at the right scope and level of detail, without irrelevant tangents. It is separate from factual correctness and groundedness.
A relevant answer responds to the actual intent. It can be fully grounded and still irrelevant if it answers a nearby question, over-explains, or omits the requested comparison. Relevance is about fit, not truth.
- State the expected scope in the rubric: direct answer, explanation, comparison or action.
- Penalise missing the requested constraint as well as adding irrelevant material.
- Judge relevance without access to the gold answer to avoid leaking correctness.
- Compare against user satisfaction signals when available.
Follow-ups you should expect
Interviewers use these to check whether the first answer was understanding or memorisation.
- 1How would you score an answer that is correct but far too long?
- 2Why judge relevance without the gold answer?
- 3When would you use user signals instead of a judge?
Say it in the interview
Relevance asks whether the answer addresses the actual question at the right scope. It can be fully grounded and still miss the point, for example by answering a nearby question or ignoring a constraint. I write the expected scope into the rubric and penalise both missing content and irrelevant tangents. I judge it without the gold answer so correctness does not leak into the relevance score.
Use this as a model, not a script. Saying it in your own words is what makes it survive follow-up pressure.
Related questions
Reading is step one
Now say it out loud.
Understanding an answer and delivering it under pressure are different skills. Rehearse this topic in a live LLM Evals mock interview and get scored feedback on depth, clarity and confidence.