LLM Evals · Turn quality into a signal
How do you prevent eval data leaking into training or prompts?
Short answer
Keep eval sets out of training corpora and prompt examples, track provenance, hash and deduplicate against training data, and hold out a private test set that is never used for iteration.
Leakage makes scores look better than reality. It can happen through fine-tuning data, few-shot examples in a prompt, retrieval corpora that include the eval answers, or sharing a dataset across teams without controls. Once a case is in the prompt, it is no longer a test.
- Store eval sets separately with access controls and provenance.
- Deduplicate eval cases against training and retrieval corpora.
- Never use eval examples as few-shot demonstrations for the same feature.
- Maintain a private holdout used only for final validation.
Follow-ups you should expect
Interviewers use these to check whether the first answer was understanding or memorisation.
- 1How would you detect leakage after the fact?
- 2Why is a private holdout useful if you already have a golden set?
- 3What is the risk of using production failures as few-shot examples?
Say it in the interview
Leakage happens when the eval case is also in training data, retrieved context or the prompt's examples. Then the score measures memorisation, not generalisation. I keep eval sets separate, deduplicate against training and retrieval corpora, and never use the same cases as few-shot demonstrations. I also keep a private holdout that is not used during iteration, so the final number is honest.
Use this as a model, not a script. Saying it in your own words is what makes it survive follow-up pressure.
Related questions
Reading is step one
Now say it out loud.
Understanding an answer and delivering it under pressure are different skills. Rehearse this topic in a live LLM Evals mock interview and get scored feedback on depth, clarity and confidence.