LLM Evals · Turn quality into a signal
How do you find and represent edge cases?
Short answer
Mine failures, boundary inputs and unusual user segments; derive cases from incident reports and domain rules. Represent them explicitly rather than hoping random sampling finds them.
Edge cases are where products break and where evals earn their keep. They are rarely common in traffic, so uniform sampling misses them. Sources include support tickets, incident reviews, domain experts, adversarial testing and boundary analysis of inputs and rules.
- Boundary inputs: empty, maximum length, malformed, mixed language.
- Rare but costly intents: refunds, safety, legal, account deletion.
- Adversarial behaviour: injection, policy probing, prompt leaks.
- Domain rules with exceptions that general examples never exercise.
Follow-ups you should expect
Interviewers use these to check whether the first answer was understanding or memorisation.
- 1How would you weight edge cases in the overall score?
- 2Where would you look for edge cases you have not thought of?
- 3How do you keep edge-case tests from overfitting?
Say it in the interview
I look for edge cases in failures, incidents, support tickets and domain rules, not in random traffic. Boundaries like empty input, maximum length, mixed language and rare high-cost intents matter more than their frequency suggests. I represent them explicitly and report them separately, because an overall average can hide a rare catastrophic failure or make a small fix look like a big win.
Use this as a model, not a script. Saying it in your own words is what makes it survive follow-up pressure.
Related questions
Reading is step one
Now say it out loud.
Understanding an answer and delivering it under pressure are different skills. Rehearse this topic in a live LLM Evals mock interview and get scored feedback on depth, clarity and confidence.