LLM Evals · Turn quality into a signal
How do prompts and eval results stay linked?
Short answer
Store prompt versions with their eval results and model settings, record the version on every trace, and re-run the suite when the prompt or model changes.
An eval result is meaningless without knowing exactly what was evaluated. Prompt text, model version, sampling parameters, retrieval configuration and judge version all affect the score. Linking them makes results reproducible and incidents diagnosable.
- Give every prompt a version and store it in version control.
- Attach prompt, model and judge versions to every eval run.
- Record the prompt version on production traces too.
- Re-run evals on model upgrades even when the prompt is unchanged.
Follow-ups you should expect
Interviewers use these to check whether the first answer was understanding or memorisation.
- 1What metadata would you attach to an eval run?
- 2How do you compare results across prompt versions?
- 3Why do model upgrades require re-running evals?
Say it in the interview
I link every eval run to the prompt version, model version, sampling settings, retrieval config and judge version. Otherwise the score cannot be reproduced or trusted. I also record the prompt version on production traces, so an incident can be tied to a specific release. When the provider updates a model, I re-run the suite even if the prompt text is identical, because behaviour can change underneath.
Use this as a model, not a script. Saying it in your own words is what makes it survive follow-up pressure.
Related questions
Reading is step one
Now say it out loud.
Understanding an answer and delivering it under pressure are different skills. Rehearse this topic in a live LLM Evals mock interview and get scored feedback on depth, clarity and confidence.