RAG & Retrieval · Ground answers in evidence
How do you deduplicate documents and chunks?
Short answer
Detect exact duplicates by hash and near-duplicates by shingling or embedding similarity. Keep one canonical copy, preserve provenance, and remove duplicates before they crowd retrieval results.
Duplicate content wastes index space and retrieval slots. Worse, it creates false confidence: five copies of the same paragraph look like five independent sources. Near-duplicates are common in versioned policies, mirrored documentation and syndicated articles.
- Exact duplicates: normalise whitespace and hash the content.
- Near-duplicates: compare shingles, MinHash or embeddings above a similarity threshold.
- Choose a canonical source and record mirrors as aliases for citations.
- Deduplicate at retrieval too, so the context window is not filled with the same passage.
Follow-ups you should expect
Interviewers use these to check whether the first answer was understanding or memorisation.
- 1How would you decide which duplicate is canonical?
- 2What is the risk of deduplicating near-identical policy versions?
- 3Where would you deduplicate: ingest or retrieval?
Say it in the interview
I hash exact duplicates and use shingles or embeddings for near-duplicates. One copy becomes canonical, and the others are recorded as aliases so citations still resolve. I also deduplicate at retrieval, because five near-identical chunks crowd out other evidence and make the answer look better supported than it is. The exception is versioned policy, where a small wording difference can matter, so I keep those distinct.
Use this as a model, not a script. Saying it in your own words is what makes it survive follow-up pressure.
Related questions
Reading is step one
Now say it out loud.
Understanding an answer and delivering it under pressure are different skills. Rehearse this topic in a live RAG & Retrieval mock interview and get scored feedback on depth, clarity and confidence.