RAG & Retrieval · Ground answers in evidence
Why does RAG still hallucinate?
Short answer
Retrieval supplies evidence but does not force the model to use it. Missing evidence, conflicting passages, weak instructions and long contexts still produce unsupported claims.
RAG reduces hallucination by giving the model relevant text, but the model can still ignore it, blend it with parametric memory, or overgeneralise from a partial passage. If retrieval misses, the model may answer anyway rather than admit the gap.
- Retrieval miss or low-quality chunk: the evidence is absent or incomplete.
- Conflicting sources: the model reconciles them into a claim neither supports.
- Weak grounding instructions: the prompt does not require citations or abstention.
- Long context: the relevant sentence is buried and attention is diluted.
- Evaluation gap: no faithfulness check, so the failure is invisible.
Follow-ups you should expect
Interviewers use these to check whether the first answer was understanding or memorisation.
- 1How would you tell a retrieval hallucination from a generation hallucination?
- 2What would you do when two sources contradict each other?
- 3How does context length affect faithfulness?
Say it in the interview
RAG gives the model evidence, but it does not compel the model to use it. It can answer from memory, blend sources, or overgeneralise a partial passage. If retrieval misses, it may still produce a confident answer instead of saying the evidence is insufficient. I treat it as a pipeline problem: improve retrieval, require citations, allow abstention, and measure faithfulness against the exact chunks supplied.
Use this as a model, not a script. Saying it in your own words is what makes it survive follow-up pressure.
Related questions
Reading is step one
Now say it out loud.
Understanding an answer and delivering it under pressure are different skills. Rehearse this topic in a live RAG & Retrieval mock interview and get scored feedback on depth, clarity and confidence.