LLM Evals · Turn quality into a signal
How do you evaluate multi-turn conversations?
Short answer
Evaluate the whole conversation: goal completion, context retention, correction handling and consistency across turns. Single-turn scores miss contradictions and forgotten constraints.
Multi-turn quality is not the average of independent turns. The model has to remember earlier constraints, resolve references, handle corrections and stay consistent. A good answer in turn three can contradict a promise made in turn one.
- Define the goal and required constraints for the whole conversation.
- Test corrections: the user changes a requirement and the model must adapt.
- Test reference resolution across turns.
- Check consistency of facts, commitments and tone.
- Use scripted user simulators for scale, but validate them against real transcripts.
Follow-ups you should expect
Interviewers use these to check whether the first answer was understanding or memorisation.
- 1How would you test that the model remembers an early constraint?
- 2What is the risk of using an LLM to simulate the user?
- 3How would you score a conversation that succeeds but contradicts itself?
Say it in the interview
I evaluate the conversation as a whole: did it achieve the goal, retain earlier constraints, resolve references and handle corrections. Averaging single-turn scores misses contradictions and forgotten requirements. I use simulated users for scale, but I make them terse, ambiguous and prone to changing their mind, because a cooperative simulator makes the system look more robust than it is. Real transcripts validate the simulation.
Use this as a model, not a script. Saying it in your own words is what makes it survive follow-up pressure.
Related questions
Reading is step one
Now say it out loud.
Understanding an answer and delivering it under pressure are different skills. Rehearse this topic in a live LLM Evals mock interview and get scored feedback on depth, clarity and confidence.