RAG & Retrieval · Ground answers in evidence
How do you control RAG cost and latency?
Short answer
Cut context size, cache repeated queries and embeddings, parallelise retrieval branches, rerank only a bounded set, and route simple questions to cheaper paths.
RAG spends money in two places: building and storing the index, and serving each query with retrieval plus generation. Serving usually dominates because every request pays for embedding, search, reranking and a long prompt.
- Reduce packed context to the passages that matter; rerank before generating.
- Cache embeddings for repeated queries and answers for repeated requests.
- Run lexical and vector retrieval in parallel.
- Use a smaller model for query rewriting or classification.
- Batch offline embedding jobs and avoid re-embedding unchanged content.
Follow-ups you should expect
Interviewers use these to check whether the first answer was understanding or memorisation.
- 1Which stage would you optimise first for a slow p95?
- 2What are the risks of caching RAG answers?
- 3How would you measure cost per answered question?
Say it in the interview
Serving cost is mostly context length plus the number of calls in the request path. I rerank before packing, send fewer passages, cache repeated queries and parallelise lexical and vector retrieval. Query rewriting can use a small model instead of the main one. I measure cost per answered question, not per API call, because retries and failures still cost money without producing value.
Use this as a model, not a script. Saying it in your own words is what makes it survive follow-up pressure.
Related questions
Reading is step one
Now say it out loud.
Understanding an answer and delivering it under pressure are different skills. Rehearse this topic in a live RAG & Retrieval mock interview and get scored feedback on depth, clarity and confidence.