LLM Evals · Turn quality into a signal
How do you handle an AI quality incident?
Short answer
Contain first, then diagnose with traces and versions, communicate honestly, and add regression cases before re-enabling. Treat quality incidents like reliability incidents with a runbook.
An AI incident might be harmful output, a data leak, a prompt injection, a sudden quality drop or runaway cost. The first priority is containment: disable the feature, restrict the tool or roll back the prompt or model. Diagnosis comes after the bleeding stops.
- Contain: feature flag off, roll back the prompt or model, revoke exposed credentials.
- Diagnose: compare traces, prompt versions, model version and retrieval state.
- Assess: scope, affected users, data exposure and regulatory obligations.
- Remediate: fix the cause, add regression cases and re-run the full suite.
- Review: document the incident and update guardrails and monitoring.
Follow-ups you should expect
Interviewers use these to check whether the first answer was understanding or memorisation.
- 1What would you do first if the feature were leaking another tenant's data?
- 2How would you decide whether to notify users?
- 3What regression cases would you add after the incident?
Say it in the interview
I contain first: turn the feature off or roll back the prompt, model or tool access. Then I diagnose with traces and version history, scope the impact and assess whether data or safety was affected. Remediation includes regression cases so the same failure cannot ship again. I do not ask the model to explain itself; the trace is the evidence. Afterwards I update monitoring and guardrails based on what the incident exposed.
Use this as a model, not a script. Saying it in your own words is what makes it survive follow-up pressure.
Related questions
Reading is step one
Now say it out loud.
Understanding an answer and delivering it under pressure are different skills. Rehearse this topic in a live LLM Evals mock interview and get scored feedback on depth, clarity and confidence.