RAG & Retrieval · Ground answers in evidence
Why do production systems use approximate nearest neighbour indexes?
Short answer
Exact similarity search is linear in the number of vectors, so it becomes too slow and expensive at scale. ANN indexes explore a smaller candidate space and typically return most true neighbours much faster.
With a million chunks and a 1024-dimension vector, brute force means a million distance computations per query. ANN structures such as HNSW, IVF and product quantisation reduce that by organising vectors so only a subset needs checking. The result is high but not perfect recall.
- Tune the recall-speed trade-off for the product; a chat assistant and a compliance search have different bars.
- Measure recall against exact search on a sample before trusting an ANN configuration.
- Index build time, memory and update behaviour differ significantly between algorithms.
- Filters and ANN interact: heavy filtering can reduce recall if not handled well.
Follow-ups you should expect
Interviewers use these to check whether the first answer was understanding or memorisation.
- 1How would you measure ANN recall in production?
- 2What happens to recall when you add a restrictive metadata filter?
- 3Would you ever use exact search at scale?
Say it in the interview
Exact search compares the query with every vector, which becomes too slow at millions of chunks. ANN indexes organise the space so only a promising subset is checked. That gives much better latency, but recall is no longer guaranteed. I tune and measure that trade-off against exact search on a sample, because a retrieval miss here is invisible later but caps answer quality.
Use this as a model, not a script. Saying it in your own words is what makes it survive follow-up pressure.
Related questions
Reading is step one
Now say it out loud.
Understanding an answer and delivering it under pressure are different skills. Rehearse this topic in a live RAG & Retrieval mock interview and get scored feedback on depth, clarity and confidence.