AI Engineering · LLM applications that ship
Offline vs online evaluation: what does each catch?
Short answer
Offline evals run a fixed dataset before release and catch regressions cheaply. Online evals measure real traffic after release and catch distribution shift, latency and user-perceived quality.
Offline evaluation is the pre-release gate: a versioned dataset, a scorer, and a pass threshold in CI. It is fast, repeatable and safe, but it only represents the cases someone thought to include. Online evaluation observes what actual users do and exposes gaps the dataset missed.
- Offline: golden sets, adversarial cases, model comparison, prompt regression tests.
- Online: thumbs up and down, task completion, escalation rate, latency and cost per request.
- Use offline to block a bad release and online to decide whether a good release is actually better.
- Sample online traffic back into the offline set so the two stay connected.
Follow-ups you should expect
Interviewers use these to check whether the first answer was understanding or memorisation.
- 1Which one would you build first for a new feature?
- 2How would you turn an online failure into an offline test case?
- 3What online metric would reveal a retrieval regression?
Say it in the interview
Offline evals are my release gate. They run the same dataset on every prompt or model change and catch regressions before users see them. Online evals tell me what the dataset missed: real query distribution, latency, escalation and satisfaction. I connect the two by sampling bad online cases back into the golden set, so each production failure becomes a regression test.
Use this as a model, not a script. Saying it in your own words is what makes it survive follow-up pressure.
Related questions
Reading is step one
Now say it out loud.
Understanding an answer and delivering it under pressure are different skills. Rehearse this topic in a live AI Engineering mock interview and get scored feedback on depth, clarity and confidence.