AI Engineering · LLM applications that ship
How do you control LLM cost in production?
Short answer
Reduce tokens, route to the cheapest capable model, cache repeated work, and cap output. Measure cost per successful task rather than cost per call.
Cost is driven by input tokens, output tokens and the model price. Long prompts with retrieved context are often the biggest line item, followed by verbose outputs. Optimising one dimension while ignoring the others usually just moves the bill.
- Trim retrieved context and conversation history to what the task needs.
- Route simple classification or extraction to a smaller model.
- Cache deterministic or low-temperature results keyed by normalised input.
- Constrain output length and avoid asking the model to restate the input.
- Track cost per resolved request, including retries and failed calls.
Follow-ups you should expect
Interviewers use these to check whether the first answer was understanding or memorisation.
- 1How would you decide which requests deserve the expensive model?
- 2What are the risks of aggressive caching?
- 3How would you attribute cost to a feature or customer?
Say it in the interview
I look at cost per successful task, not per API call. The biggest levers are shorter context, shorter outputs, routing easy requests to a smaller model, and caching repeated work. I also count retries and failures, because a cheap model that fails twice is not cheap. Before adding a model call I ask whether deterministic code can do the job, since removing a call is the most reliable saving.
Use this as a model, not a script. Saying it in your own words is what makes it survive follow-up pressure.
Related questions
Reading is step one
Now say it out loud.
Understanding an answer and delivering it under pressure are different skills. Rehearse this topic in a live AI Engineering mock interview and get scored feedback on depth, clarity and confidence.