LLM Evals · Turn quality into a signal
How do you run evals in CI without slowing the team?
Short answer
Split a fast smoke suite for every change from a slower comprehensive suite for release candidates, cache results, parallelise cases and fail only on regressions that matter.
A full judge run on every commit is too slow and expensive. A tiered pipeline keeps feedback fast: deterministic checks and a small critical suite on every pull request, broader evals before release, and a nightly run for drift and long-tail cases.
- Smoke: schema validity, critical-case pass rate, a small judge sample.
- Release: full golden set, category breakdown, latency and cost.
- Nightly: drift checks, larger samples, adversarial and long-tail cases.
- Cache unchanged cases and parallelise independent evaluations.
- Show per-case diffs so reviewers can see what changed.
Follow-ups you should expect
Interviewers use these to check whether the first answer was understanding or memorisation.
- 1Which checks belong on every pull request?
- 2How would you handle flaky judge scores in CI?
- 3What would you do when the full suite takes too long?
Say it in the interview
I tier the suite. Every pull request gets deterministic checks and a small critical set, release candidates get the full golden set with category breakdowns, and a nightly job handles drift and long-tail cases. I cache unchanged results and parallelise. CI should fail on critical regressions, not on small average noise, otherwise people start ignoring the signal.
Use this as a model, not a script. Saying it in your own words is what makes it survive follow-up pressure.
Related questions
Reading is step one
Now say it out loud.
Understanding an answer and delivering it under pressure are different skills. Rehearse this topic in a live LLM Evals mock interview and get scored feedback on depth, clarity and confidence.