AI Engineering · LLM applications that ship
How should an LLM feature degrade when the model fails?
Short answer
Define fallbacks per failure type: cached or template answers, a smaller model, a human handoff, or an honest error. Degrade the feature, not the truthfulness of the response.
Models time out, rate-limit, return invalid output and occasionally produce unsafe content. A fallback plan decides what the user sees in each case. The worst outcome is a fabricated answer presented with the same confidence as a normal one.
- Transient provider failure: retry briefly, then use a smaller model or cached answer.
- Invalid output: repair once, then fall back to a template or clarification question.
- Low retrieval confidence: say the evidence is insufficient and offer a human path.
- Safety block: explain the limit without exposing internal policy details.
Follow-ups you should expect
Interviewers use these to check whether the first answer was understanding or memorisation.
- 1Which failure should never silently fall back to a generic answer?
- 2How would you test fallbacks in staging?
- 3What would you show the user during a long provider outage?
Say it in the interview
I define fallbacks by failure type. For a transient provider error I retry briefly, then switch to a smaller model or a cached answer. For invalid or low-confidence output I ask for clarification or hand off to a human rather than guessing. The rule is to degrade the feature, not the truthfulness. I also test the fallback paths, because an untested fallback tends to be broken exactly when production needs it.
Use this as a model, not a script. Saying it in your own words is what makes it survive follow-up pressure.
Related questions
Reading is step one
Now say it out loud.
Understanding an answer and delivering it under pressure are different skills. Rehearse this topic in a live AI Engineering mock interview and get scored feedback on depth, clarity and confidence.