LLM Evals · Turn quality into a signal
Who owns evals on a product team?
Short answer
The feature team owns evals, with domain experts contributing labels and quality criteria. Central platform support is useful, but quality cannot be outsourced away from the people shipping the feature.
Evals encode product judgement: what counts as a good answer, which failures are unacceptable, and what trade-offs are allowed. That knowledge sits with the feature team and domain experts, not only with an ML platform group. Ownership means maintaining the dataset, reviewing failures and gating releases.
- Feature engineer: harness, scoring, CI integration and reporting.
- Domain expert: labelling guidelines, edge cases and quality review.
- Product: acceptable trade-offs, thresholds and abstention policy.
- Platform: shared tooling, storage and best practices.
Follow-ups you should expect
Interviewers use these to check whether the first answer was understanding or memorisation.
- 1How would you split ownership between product and engineering?
- 2What happens when the eval owner leaves the team?
- 3How do you keep domain experts engaged without burning them out?
Say it in the interview
The feature team owns the evals because they own the product judgement. Engineering maintains the harness and CI, domain experts write the labelling guidelines and review hard cases, and product decides the acceptable trade-offs and thresholds. A central team can provide tooling, but it cannot own quality for every feature. I also make sure there is a named owner, because eval suites decay fast without one.
Use this as a model, not a script. Saying it in your own words is what makes it survive follow-up pressure.
Related questions
Reading is step one
Now say it out loud.
Understanding an answer and delivering it under pressure are different skills. Rehearse this topic in a live LLM Evals mock interview and get scored feedback on depth, clarity and confidence.