AI Engineering · LLM applications that ship
How do you choose a model for a feature?
Short answer
Choose on eval quality for the actual task, then weigh latency, cost, context size and operational constraints. Re-evaluate when models or requirements change.
Benchmark leaderboards are weak evidence for a specific product. The right model is the cheapest one that clears your quality bar on your data, with acceptable latency and reliability. Different steps in the same feature may deserve different models.
- Run the same eval set across candidate models, including edge cases.
- Measure p95 latency and cost per successful task, not just quality.
- Check context limits, structured-output support, tool calling and data-retention terms.
- Use a small model for routing or extraction and a larger one for reasoning-heavy steps.
- Keep the model behind a config so switching does not require a code rewrite.
Follow-ups you should expect
Interviewers use these to check whether the first answer was understanding or memorisation.
- 1What would make you switch models after launch?
- 2How would you compare two models fairly?
- 3When is a multi-model setup worth the complexity?
Say it in the interview
I choose on the task, not the leaderboard. I run my own eval set across the candidates, then compare latency, cost and operational fit. The cheapest model that clears the bar wins. I often use a small model for routing or extraction and a larger one only for the reasoning step. I keep the model configurable and re-evaluate when the provider releases an update or the product's requirements change.
Use this as a model, not a script. Saying it in your own words is what makes it survive follow-up pressure.
Related questions
Reading is step one
Now say it out loud.
Understanding an answer and delivering it under pressure are different skills. Rehearse this topic in a live AI Engineering mock interview and get scored feedback on depth, clarity and confidence.