LLM Evals · Turn quality into a signal
How do you A/B test an LLM feature?
Short answer
Randomise users, define a primary outcome before launch, keep prompts and models fixed within the experiment, and watch guardrail metrics such as latency, cost and safety.
An A/B test answers whether a change improves the product, not whether it improves an offline score. The primary metric must be tied to user value: task completion, resolution rate or retention. Offline evals decide what is safe to test; the experiment decides whether it actually helps.
- Pre-register the primary metric, sample size and stopping rule.
- Randomise at a stable unit such as user or session.
- Change one thing at a time where possible.
- Monitor latency, cost, refusal rate and safety as guardrails.
- Segment results by intent to catch heterogeneous effects.
Follow-ups you should expect
Interviewers use these to check whether the first answer was understanding or memorisation.
- 1What primary metric would you choose for a support assistant?
- 2How would you handle a quality win that increases cost?
- 3Why randomise by user rather than by request?
Say it in the interview
I use offline evals to decide what is safe to test, then an A/B test to decide whether it actually helps. I pre-register the primary metric and sample size, randomise by user or session, and keep the model and prompt fixed within each arm. I watch latency, cost and safety as guardrails. Engagement is noisy, so I do not call a winner from a short test on a small sample.
Use this as a model, not a script. Saying it in your own words is what makes it survive follow-up pressure.
Related questions
Reading is step one
Now say it out loud.
Understanding an answer and delivering it under pressure are different skills. Rehearse this topic in a live LLM Evals mock interview and get scored feedback on depth, clarity and confidence.