AI Engineering · LLM applications that ship
How do you design an LLM latency budget?
Short answer
Set a user-facing target, then allocate it across retrieval, model time to first token, generation and post-processing. Stream output and cut work that does not affect perceived speed.
Users experience time to first token and total time differently. Streaming makes a two-second first token feel responsive if words appear steadily. A latency budget makes those trade-offs explicit before you add another retrieval hop or guardrail call.
- Measure p50 and p95, not just the average; tail latency drives complaints.
- Parallelise independent work such as retrieval branches and metadata lookups.
- Cap output length and avoid unnecessary model calls in the request path.
- Use smaller or faster models for routing and classification, and reserve the large model for the hard step.
Follow-ups you should expect
Interviewers use these to check whether the first answer was understanding or memorisation.
- 1Which latency number would you put in front of users?
- 2How does streaming change perceived performance?
- 3What would you cut first if the budget were exceeded?
Say it in the interview
I start from the user-visible target and split it across retrieval, first token, generation and any post-processing. I care about p95 because the tail is what people complain about. Independent work runs in parallel, and I stream tokens so perceived latency is lower. If I am over budget, I cut model calls and output length first, then consider a smaller model for routing rather than degrading the core answer.
Use this as a model, not a script. Saying it in your own words is what makes it survive follow-up pressure.
Related questions
Reading is step one
Now say it out loud.
Understanding an answer and delivering it under pressure are different skills. Rehearse this topic in a live AI Engineering mock interview and get scored feedback on depth, clarity and confidence.