LLM Evals · Turn quality into a signal
Should latency and cost be part of evals?
Short answer
Yes. A quality improvement that doubles p95 latency or cost per task may not be shippable. Track them alongside quality so trade-offs are explicit.
Quality is not the only product constraint. Users abandon slow experiences, and unit economics decide whether a feature can scale. If latency and cost are not in the eval report, they get discovered after launch or not at all.
- Measure p50 and p95 latency at the user-visible level.
- Track cost per successful task, including retries and judges.
- Compare against the product's budget, not only the previous version.
- Report quality, latency and cost together so trade-offs are visible.
Follow-ups you should expect
Interviewers use these to check whether the first answer was understanding or memorisation.
- 1What latency budget would you set for a chat feature?
- 2How would you compare two models with different cost-quality profiles?
- 3Why measure at the user-visible level rather than per API call?
Say it in the interview
I include latency and cost because a quality win that makes the feature too slow or too expensive is not a win. I measure p95 at the user-visible level and cost per successful task, including retries. Then I report quality, latency and cost together so the trade-off is explicit. An offline quality score alone can reward a model that nobody can afford to run.
Use this as a model, not a script. Saying it in your own words is what makes it survive follow-up pressure.
Related questions
Reading is step one
Now say it out loud.
Understanding an answer and delivering it under pressure are different skills. Rehearse this topic in a live LLM Evals mock interview and get scored feedback on depth, clarity and confidence.