AI Engineering · LLM applications that ship
How do you handle rate limits, retries and timeouts?
Short answer
Retry only transient failures with exponential backoff and jitter, respect provider rate-limit headers, cap attempts, and make non-idempotent operations safe before retrying.
LLM APIs fail in several ways: rate limits, timeouts, overloaded providers and malformed responses. Blind retries amplify load and can duplicate side effects. A retry policy should distinguish transient errors from permanent ones and know which operations are safe to repeat.
- Retry 429s and 5xx errors with exponential backoff plus jitter; do not retry 4xx validation errors.
- Cap attempts and total time, then fall back to a degraded path.
- Use idempotency keys or check-before-write for tool actions that change state.
- Queue and shed load when the provider is saturated rather than retrying in the request path.
Follow-ups you should expect
Interviewers use these to check whether the first answer was understanding or memorisation.
- 1How would you make a retried payment tool call safe?
- 2What would you show the user while retries are happening?
- 3How do provider rate limits change your architecture?
Say it in the interview
I retry only errors that are actually transient, with exponential backoff and jitter, and I cap both attempts and total time. Validation errors fail fast. For tools that change state, I need idempotency or a check before retrying, otherwise a retry can duplicate the action. If the provider is saturated, I would rather queue or shed load and show a clear degraded state than hammer the API from every request.
Use this as a model, not a script. Saying it in your own words is what makes it survive follow-up pressure.
Related questions
Reading is step one
Now say it out loud.
Understanding an answer and delivering it under pressure are different skills. Rehearse this topic in a live AI Engineering mock interview and get scored feedback on depth, clarity and confidence.