LLM Evals · Turn quality into a signal
Why do LLM features need evaluations?
Short answer
LLM behaviour is variable and changes with prompts, models and data. Evals turn quality into a measurable signal so regressions are caught before users see them.
Ordinary unit tests assert exact values. LLM features produce a distribution of plausible outputs, so exact assertions either fail constantly or test nothing. Evaluations define representative inputs and a scoring method, then compare versions on the same basis.
- A prompt edit can fix one case and quietly break three others.
- Provider model updates change behaviour without any code change.
- Retrieval, tool and policy changes all affect answer quality.
- Without evals, the first regression signal is a user complaint.
Follow-ups you should expect
Interviewers use these to check whether the first answer was understanding or memorisation.
- 1What would you do if the team has no labelled data?
- 2How many eval cases are enough to start?
- 3Who should own evals on a product team?
Say it in the interview
LLM features do not fail like deterministic code. The output varies, and a prompt or model change can improve one case while breaking another. Evals give me a repeatable way to compare versions on the same inputs, so regressions are caught before users report them. I start with real queries and known failures rather than inventing examples, then grow the set over time.
Use this as a model, not a script. Saying it in your own words is what makes it survive follow-up pressure.
Related questions
Reading is step one
Now say it out loud.
Understanding an answer and delivering it under pressure are different skills. Rehearse this topic in a live LLM Evals mock interview and get scored feedback on depth, clarity and confidence.