AI Engineering · LLM applications that ship
Where does RAG fail in production?
Short answer
RAG fails in four places: ingestion, retrieval, context assembly and generation. Each has distinct symptoms, so you need stage-level traces to know which one broke.
A wrong answer is not enough information to debug RAG. The same symptom can come from a parser dropping a table, a query embedding that misses the right section, a context window packed with duplicates, or a model ignoring the evidence it was given.
- Ingestion: broken parsing, stale documents, missing permissions metadata, duplicated chunks.
- Retrieval: vocabulary mismatch, over-filtering, index drift, query ambiguity.
- Assembly: too many chunks, lost ordering, truncation that cuts the evidence sentence.
- Generation: unsupported claims, ignored citations, answering from parametric memory.
Follow-ups you should expect
Interviewers use these to check whether the first answer was understanding or memorisation.
- 1How would you reproduce a bad answer from a user report?
- 2Which failure mode is hardest to detect automatically?
- 3What would you fix first if retrieval recall were high but answers were wrong?
Say it in the interview
I separate RAG failures by stage. Ingestion can drop structure or serve stale documents, retrieval can miss the right passage, assembly can bury it in noise, and generation can ignore it. The symptom looks the same to a user, so I log retrieved chunk IDs and scores on every request. That trace tells me whether to fix parsing, search, prompt packing or the model instructions instead of guessing.
Use this as a model, not a script. Saying it in your own words is what makes it survive follow-up pressure.
Related questions
Reading is step one
Now say it out loud.
Understanding an answer and delivering it under pressure are different skills. Rehearse this topic in a live AI Engineering mock interview and get scored feedback on depth, clarity and confidence.