LLM Evals · Turn quality into a signal
What belongs in an eval harness?
Short answer
Dataset versioning, deterministic checks, judge integration, reproducible configuration, per-case reporting, cost and latency tracking, and a baseline comparison.
The harness is the infrastructure that makes evaluation repeatable. It should run the same cases against a versioned configuration, apply deterministic and model-based checks, and produce a report that shows what changed. Without it, evals become a collection of notebooks that nobody trusts.
- Versioned datasets and configuration, with provenance.
- Pluggable scorers: exact, schema, judge and human.
- Parallel execution with retries and cost controls.
- Per-case results, category breakdowns and baseline diffs.
- Trace capture so a failure can be reproduced.
- Export to CI and dashboards.
Follow-ups you should expect
Interviewers use these to check whether the first answer was understanding or memorisation.
- 1What would you build first if you had one week?
- 2How would you store per-case results over time?
- 3How do you keep the harness from becoming a second product?
Say it in the interview
A harness needs versioned datasets and configuration, pluggable scorers, per-case reporting and a baseline diff. It also needs cost and latency tracking and trace capture so failures are reproducible. I would build the smallest version that can run a fixed set, show what changed and integrate with CI. The measure of a good harness is that adding a regression case and re-running is easy, not that it supports every metric imaginable.
Use this as a model, not a script. Saying it in your own words is what makes it survive follow-up pressure.
Related questions
Reading is step one
Now say it out loud.
Understanding an answer and delivering it under pressure are different skills. Rehearse this topic in a live LLM Evals mock interview and get scored feedback on depth, clarity and confidence.