LLM Evals · Turn quality into a signal
When should you generate synthetic eval data?
Short answer
Use synthetic data to fill coverage gaps, vary phrasing or simulate rare scenarios, then validate it and mix it with real cases. Never let synthetic data become the only ground truth.
Synthetic data is cheap and fast, which makes it tempting to overuse. A model generating both the question and the expected answer can bake in its own assumptions and produce cases that look diverse but test the same thing. It is most useful for augmentation and stress testing.
- Use real cases as the anchor and synthetic variants for coverage.
- Generate inputs, but have humans or evidence-based checks validate labels.
- Watch for duplication and stylistic artifacts from the generator.
- Hold out a purely real test set to detect synthetic-data inflation.
Follow-ups you should expect
Interviewers use these to check whether the first answer was understanding or memorisation.
- 1How would you detect that synthetic cases are too easy?
- 2What should never be synthetic in an eval set?
- 3How do you keep synthetic data from leaking into prompts?
Say it in the interview
I use synthetic data to fill gaps and create variations, but I anchor the set in real cases. The trap is letting a model generate both the question and the expected answer, which produces cases that look diverse but share the same blind spots. I validate labels with humans or evidence, keep a purely real holdout, and watch for duplication and stylistic artifacts.
Use this as a model, not a script. Saying it in your own words is what makes it survive follow-up pressure.
Related questions
Reading is step one
Now say it out loud.
Understanding an answer and delivering it under pressure are different skills. Rehearse this topic in a live LLM Evals mock interview and get scored feedback on depth, clarity and confidence.