LLM Evals · Turn quality into a signal
Offline vs online evaluation?
Short answer
Offline evals run a fixed dataset before release and catch regressions cheaply. Online evals measure real traffic after release and catch distribution shift and user-perceived quality.
Offline evaluation is a gate: versioned dataset, scorer, threshold. It is repeatable and safe, but limited to cases someone included. Online evaluation observes actual usage and can reveal intents, phrasings and failures the dataset missed.
- Offline: golden sets, adversarial cases, model comparison, prompt regression.
- Online: satisfaction signals, escalation rate, task completion, latency.
- Use offline to block a bad release and online to judge real-world improvement.
- Feed sampled online failures back into the offline set.
Follow-ups you should expect
Interviewers use these to check whether the first answer was understanding or memorisation.
- 1Which would you build first for a brand-new feature?
- 2How do you turn an online complaint into an offline case?
- 3What online metric would reveal a retrieval regression?
Say it in the interview
Offline evals are the release gate: a fixed dataset and scorer that I can run on every change. Online evals show what the dataset missed, because real users ask things we did not anticipate. I use offline to prevent regressions and online to decide whether the feature is genuinely better in practice. Then I sample bad online cases back into the golden set so the gate improves.
Use this as a model, not a script. Saying it in your own words is what makes it survive follow-up pressure.
Related questions
Reading is step one
Now say it out loud.
Understanding an answer and delivering it under pressure are different skills. Rehearse this topic in a live LLM Evals mock interview and get scored feedback on depth, clarity and confidence.