LLM Evals · Turn quality into a signal
How do you build a golden set from scratch?
Short answer
Collect real inputs, cluster them by intent, sample across clusters and difficulty, write expected behaviour, label in rounds and review disagreements before freezing a version.
Start with production logs or user interviews, not imagination. Cluster queries by intent so the set is not dominated by one common case. Then choose examples deliberately: typical, difficult, ambiguous and out of scope. Write the expected behaviour before running the model.
- Gather 100–300 real inputs and group them by intent.
- Select a balanced subset, oversampling rare but important cases.
- Write expected behaviour or a rubric per case.
- Label in two rounds and discuss disagreements.
- Freeze a versioned baseline and record its provenance.
Follow-ups you should expect
Interviewers use these to check whether the first answer was understanding or memorisation.
- 1How would you handle intents with no real examples yet?
- 2How many cases would you put in the first version?
- 3What metadata should each case carry?
Say it in the interview
I start from real inputs and cluster them by intent so the set is not skewed by the most common query. Then I pick a balanced subset, oversampling rare but important cases, and write expected behaviour before running the model. I label in two rounds and discuss disagreements. I freeze a versioned baseline and record where each case came from, so later changes are comparable.
Use this as a model, not a script. Saying it in your own words is what makes it survive follow-up pressure.
Related questions
Reading is step one
Now say it out loud.
Understanding an answer and delivering it under pressure are different skills. Rehearse this topic in a live LLM Evals mock interview and get scored feedback on depth, clarity and confidence.