AI Engineering · LLM applications that ship
When and how should you cache LLM calls?
Short answer
Cache when inputs repeat and staleness is acceptable, keyed by a normalised prompt, model and parameters. Never cache personalised or permission-sensitive output without scoping the key.
Caching cuts latency and cost for repeated questions, but LLM output depends on the full request: system prompt, model version, temperature, retrieved context and user permissions. A cache key that ignores any of these can serve the wrong answer to the wrong person.
- Cache exact repeats for FAQ, classification and templated extraction.
- Include model version, prompt version and relevant context hashes in the key.
- Respect permissions and personalisation in the key or do not cache at all.
- Set a time-to-live based on how often the underlying knowledge changes.
- Use provider prompt caching for long stable prefixes where available.
Follow-ups you should expect
Interviewers use these to check whether the first answer was understanding or memorisation.
- 1What would you include in a cache key for a RAG answer?
- 2How would you invalidate the cache when documents change?
- 3When is semantic caching too risky?
Say it in the interview
I cache only when the same request is likely to repeat and stale answers are acceptable. The key has to include the model, prompt version, parameters, relevant context and any permission scope, otherwise one user could receive another user's answer. I set a TTL tied to how often the source changes. Semantic caching can save more, but I gate it with a similarity threshold and test it, because near-matches can be confidently wrong.
Use this as a model, not a script. Saying it in your own words is what makes it survive follow-up pressure.
Related questions
Reading is step one
Now say it out loud.
Understanding an answer and delivering it under pressure are different skills. Rehearse this topic in a live AI Engineering mock interview and get scored feedback on depth, clarity and confidence.