RAG & Retrieval · Ground answers in evidence
When is RAG the wrong approach?
Short answer
RAG is wrong when the task needs reasoning over structured data, exact arithmetic, real-time actions or a small fixed knowledge set that fits in the prompt. Use tools, SQL or deterministic code instead.
RAG is a retrieval pattern, not a universal answer. If the user asks for the total of their last five invoices, retrieval may find the invoices but the model still should not do the arithmetic. If the answer is a live account balance, a tool call is the right source.
- Structured aggregation: query the database with SQL, then let the model explain the result.
- Real-time state: call the API that owns the state.
- Small, stable instruction set: put it in the prompt.
- Exact policy logic: encode it in code and use the model for language, not judgement.
Follow-ups you should expect
Interviewers use these to check whether the first answer was understanding or memorisation.
- 1How would you answer a question that needs both documents and live data?
- 2When would a SQL tool beat a vector index?
- 3What would you do if the corpus is only a few pages?
Say it in the interview
RAG is wrong when the answer is not really in unstructured text. If the user needs a sum, a live balance or an action, I call a tool or query the database. If the knowledge is small and stable, it belongs in the prompt. RAG is for finding relevant passages, not for doing arithmetic or replacing a system of record. Often the best design combines a deterministic query with retrieved context around it.
Use this as a model, not a script. Saying it in your own words is what makes it survive follow-up pressure.
Related questions
Reading is step one
Now say it out loud.
Understanding an answer and delivering it under pressure are different skills. Rehearse this topic in a live RAG & Retrieval mock interview and get scored feedback on depth, clarity and confidence.