RAG & Retrieval · Ground answers in evidence
How do you pick chunk size and overlap?
Short answer
Match chunk size to the granularity of questions and the embedding model, then tune on retrieval evals. Overlap protects boundary sentences but increases index size and duplicates.
Small chunks retrieve precisely but may lack context. Large chunks carry context but blur the embedding and waste prompt space. There is no universal number: an API reference and a legal contract need different granularity.
- Start with the document's natural unit, such as a section or a few paragraphs.
- Measure recall and context precision on a labelled query set.
- Use overlap only where sentences cross boundaries; excessive overlap creates near-duplicates.
- Account for the reranker and prompt budget when choosing the candidate size.
Follow-ups you should expect
Interviewers use these to check whether the first answer was understanding or memorisation.
- 1How would you detect that chunks are too large?
- 2What is the cost of too much overlap?
- 3Would you use different chunk sizes for different document types?
Say it in the interview
I start from the document's natural structure and then tune on retrieval evals. Small chunks are precise but can lose context; large chunks keep context but dilute the embedding and eat the prompt budget. Overlap helps with sentences that straddle a boundary, but too much creates near-duplicates and inflates the index. Since changing size means re-embedding, I test on a copy and compare recall and context precision.
Use this as a model, not a script. Saying it in your own words is what makes it survive follow-up pressure.
Related questions
Reading is step one
Now say it out loud.
Understanding an answer and delivering it under pressure are different skills. Rehearse this topic in a live RAG & Retrieval mock interview and get scored feedback on depth, clarity and confidence.