RAG & Retrieval · Ground answers in evidence
How do you avoid returning near-duplicate results?
Short answer
Use maximal marginal relevance or similarity thresholds to balance relevance against novelty, and deduplicate at ingest and retrieval. A context window full of one source adds no new evidence.
Top-k retrieval often returns several chunks that say the same thing. That looks like strong support but leaves the model without complementary evidence. Diversity-aware selection picks results that are relevant but not redundant.
- MMR scores each candidate by relevance minus similarity to already selected results.
- Group chunks by source and cap how many come from one document.
- Deduplicate near-identical text before ranking.
- Tune the relevance-diversity balance on downstream answer quality, not just ranking metrics.
Follow-ups you should expect
Interviewers use these to check whether the first answer was understanding or memorisation.
- 1How would MMR change the context you send to the model?
- 2When would diversity hurt answer quality?
- 3Would you apply diversity before or after reranking?
Say it in the interview
If the top results are all near-duplicates, the model gets repetition instead of coverage. I use MMR or per-source caps to keep results relevant but complementary. I also deduplicate at ingest and retrieval. The balance matters: too much diversity can push out the strongest passage, so I tune it against final answer quality rather than a ranking metric alone.
Use this as a model, not a script. Saying it in your own words is what makes it survive follow-up pressure.
Related questions
Reading is step one
Now say it out loud.
Understanding an answer and delivering it under pressure are different skills. Rehearse this topic in a live RAG & Retrieval mock interview and get scored feedback on depth, clarity and confidence.