RAG & Retrieval · Ground answers in evidence
What do you monitor in a production RAG system?
Short answer
Monitor ingestion lag, retrieval health, answer quality and cost: zero-result rate, score distributions, citation validity, faithfulness samples, latency and token spend.
RAG can degrade without throwing an error. A failed ingestion job, a stale index or a shift in user queries all produce plausible answers that are quietly worse. Monitoring needs both operational signals and quality samples.
- Ingestion: failed documents, indexing lag, parser error rate, stale sources.
- Retrieval: zero-result rate, low-score rate, filter rejection rate, index size.
- Generation: citation validity, abstention rate, sampled faithfulness scores.
- Product: thumbs-down rate, escalation rate, task completion.
- Operations: p95 latency, cost per request, provider error and retry rates.
Follow-ups you should expect
Interviewers use these to check whether the first answer was understanding or memorisation.
- 1Which alert would you create first?
- 2How would you sample answers for human review?
- 3How do you connect a user complaint to a trace?
Say it in the interview
I monitor the pipeline, not just uptime. On ingestion I watch failures, lag and stale sources. On retrieval I watch zero-result and low-score rates. On generation I sample faithfulness and check citation validity, plus abstention rate. Then I track p95 latency, cost and user signals like thumbs-down and escalation. The useful alerts are distribution shifts, because a broken index often looks fine at the HTTP layer.
Use this as a model, not a script. Saying it in your own words is what makes it survive follow-up pressure.
Related questions
Reading is step one
Now say it out loud.
Understanding an answer and delivering it under pressure are different skills. Rehearse this topic in a live RAG & Retrieval mock interview and get scored feedback on depth, clarity and confidence.