LLM Evals · Turn quality into a signal
What do you monitor after launch?
Short answer
Monitor quality proxies, safety events, latency, cost and failure rates, and sample traces for human review. Watch for distribution shift, not just uptime.
After launch the input distribution changes, providers update models and retrieval corpora go stale. A feature that passed offline evals can degrade without throwing errors. Online monitoring is how you notice before users escalate.
- Quality: thumbs-down rate, edit or retry rate, escalation and abandonment.
- Safety: policy triggers, refusals and reported incidents.
- Operations: p95 latency, error rate, retry rate and cost per task.
- Data: query distribution, zero-result rate and retrieval score drift.
- Review: a sampled queue of traces judged by humans or a calibrated judge.
Follow-ups you should expect
Interviewers use these to check whether the first answer was understanding or memorisation.
- 1Which online signal would you trust most?
- 2How would you detect a silent model update?
- 3How large a sample do you need for human review?
Say it in the interview
I monitor quality proxies, safety, latency, cost and retrieval health, plus a sampled review queue. The important thing is changes relative to a baseline, because slow degradation does not cross a fixed threshold. I also watch query distribution and zero-result rates, which often move before user complaints. A silent provider model update is exactly the kind of thing only online monitoring and version tracking will catch.
Use this as a model, not a script. Saying it in your own words is what makes it survive follow-up pressure.
Related questions
Reading is step one
Now say it out loud.
Understanding an answer and delivering it under pressure are different skills. Rehearse this topic in a live LLM Evals mock interview and get scored feedback on depth, clarity and confidence.