LLM Evals · Turn quality into a signal
How do you sample production traffic for evals?
Short answer
Sample across intents, users and outcomes rather than uniformly, stratify for rare cases, redact sensitive data and track the sampling window so the set reflects current behaviour.
Uniform random sampling mirrors volume but under-represents rare failures. Stratified sampling by intent, model version, locale and outcome gives better coverage. The other risk is privacy: production data must be redacted and handled under the same policy as the feature itself.
- Stratify by intent, user segment, language and success or failure.
- Oversample low-frequency but high-risk categories.
- Redact or synthesise PII before the case enters the eval set.
- Record the sampling window and refresh periodically to track drift.
Follow-ups you should expect
Interviewers use these to check whether the first answer was understanding or memorisation.
- 1How would you sample to catch rare failures?
- 2What privacy steps are required before storing eval cases?
- 3How often would you refresh the sample?
Say it in the interview
I sample by intent and outcome rather than uniformly, because random sampling hides rare but serious failures. I oversample high-risk categories and record the time window. Privacy matters too, so I redact or synthesise personal data before a case enters the set. I refresh the sample periodically, because an old snapshot cannot reveal distribution drift, which is one of the main things online-to-offline sampling is for.
Use this as a model, not a script. Saying it in your own words is what makes it survive follow-up pressure.
Related questions
Reading is step one
Now say it out loud.
Understanding an answer and delivering it under pressure are different skills. Rehearse this topic in a live LLM Evals mock interview and get scored feedback on depth, clarity and confidence.