LLM Evals · Turn quality into a signal
What should an evaluation suite cover?
Short answer
Cover normal usage, edge cases, adversarial inputs, out-of-scope requests, format requirements and multi-turn behaviour. Coverage should follow product risk, not equal counts per category.
A suite that only covers happy paths gives false confidence. The failures that reach production usually come from ambiguity, missing context, unusual inputs or users pushing the system outside its intended scope. Coverage is a risk map, not a checklist of equal buckets.
- Common intents and phrasings, including short and messy inputs.
- Edge cases: empty input, very long input, multiple languages, typos.
- Adversarial: prompt injection, policy probing, attempts to extract secrets.
- Out of scope: requests the feature should decline or hand off.
- Behavioural: format, citation, refusal and multi-turn consistency.
Follow-ups you should expect
Interviewers use these to check whether the first answer was understanding or memorisation.
- 1How would you decide how many cases each category needs?
- 2What coverage gap most often causes incidents?
- 3How do you test refusal behaviour without over-blocking?
Say it in the interview
I map coverage to risk. That means common intents, but also messy inputs, edge cases, adversarial prompts and out-of-scope requests. I include format, citation and refusal behaviour, plus multi-turn consistency if the product is conversational. After an incident, the first question is what category was missing. Coverage should follow where failure would hurt, not an equal number of cases per bucket.
Use this as a model, not a script. Saying it in your own words is what makes it survive follow-up pressure.
Related questions
Reading is step one
Now say it out loud.
Understanding an answer and delivering it under pressure are different skills. Rehearse this topic in a live LLM Evals mock interview and get scored feedback on depth, clarity and confidence.