RAG & Retrieval · Ground answers in evidence
What is the trade-off in embedding dimensions?
Short answer
Higher dimensions can capture more nuance but cost more storage and slow similarity search. Lower dimensions are cheaper and often sufficient, especially with matryoshka or reduced-precision vectors.
Every chunk stores one vector. At a million chunks, the difference between 384 and 1536 dimensions is gigabytes of memory and a measurable search-latency change. Dimension reduction is a real lever, as long as retrieval quality holds on your evals.
- Measure recall at the full dimension, then at reduced dimensions.
- Some models support matryoshka truncation, where prefixes remain useful.
- Quantisation reduces memory but can cost a little precision.
- Storage and index type interact: HNSW keeps vectors in memory.
Follow-ups you should expect
Interviewers use these to check whether the first answer was understanding or memorisation.
- 1How would you test whether 256 dimensions are enough?
- 2What does quantisation trade away?
- 3Why does dimension affect index memory so strongly?
Say it in the interview
More dimensions can preserve more distinction, but every chunk pays for it in storage and search time. I measure recall at the full size and then at a reduced size, and I only keep the reduction if quality holds. Some models support matryoshka embeddings, where a prefix is still meaningful, which makes truncation safer. Quantisation is the other lever, but it can cost precision, so it also needs an eval.
Use this as a model, not a script. Saying it in your own words is what makes it survive follow-up pressure.
Related questions
Reading is step one
Now say it out loud.
Understanding an answer and delivering it under pressure are different skills. Rehearse this topic in a live RAG & Retrieval mock interview and get scored feedback on depth, clarity and confidence.