AI Engineering · LLM applications that ship
How do you measure RAG quality?
Short answer
Measure retrieval and generation separately: recall and context precision for retrieval, faithfulness and answer relevance for generation. End-to-end scores alone cannot tell you what to fix.
RAG has two failure surfaces. If the right evidence is never retrieved, the generator cannot succeed. If it is retrieved but ignored, the generator is at fault. Separate metrics localise the problem before you change anything.
- Context recall: was the passage needed for the answer present in the retrieved set?
- Context precision: how much of the retrieved context was actually relevant?
- Faithfulness: is every claim supported by the provided context?
- Answer relevance: does the answer address the question asked?
- Track latency and cost alongside quality so improvements stay shippable.
Follow-ups you should expect
Interviewers use these to check whether the first answer was understanding or memorisation.
- 1Which metric would you alert on in production?
- 2How do you build a retrieval eval set without exhaustive labelling?
- 3What would you change if recall is high but faithfulness is low?
Say it in the interview
I split the measurement. For retrieval I track whether the needed passage was in the candidate set and how much of the context was relevant. For generation I track faithfulness to the evidence and whether the answer actually addresses the question. That split tells me whether to work on indexing, query rewriting, reranking or the prompt. If recall is high and faithfulness is low, the model is ignoring good evidence, so I change the instructions and citation requirements.
Use this as a model, not a script. Saying it in your own words is what makes it survive follow-up pressure.
Related questions
Reading is step one
Now say it out loud.
Understanding an answer and delivering it under pressure are different skills. Rehearse this topic in a live AI Engineering mock interview and get scored feedback on depth, clarity and confidence.