LLM Evals · Turn quality into a signal
What is trajectory evaluation?
Short answer
Trajectory evaluation scores the sequence of steps an agent took: tool choices, arguments, observations, recovery and efficiency, not only the final answer.
A trajectory is the full record of an agent's run: reasoning summaries, tool calls, results and state changes. Evaluating it means checking whether each step was justified and whether the path stayed within policy and budget. Some steps are strictly required; others are acceptable variations.
- Define required checkpoints and forbidden actions, not one exact path.
- Score tool selection, argument validity, error handling and efficiency.
- Use deterministic checks for tool arguments and side effects.
- Use a judge for qualitative trajectory qualities, calibrated against humans.
- Sample multiple runs because trajectories vary.
Follow-ups you should expect
Interviewers use these to check whether the first answer was understanding or memorisation.
- 1How would you handle several valid paths to the same outcome?
- 2What would you check deterministically versus with a judge?
- 3How would you score recovery from a failed tool call?
Say it in the interview
Trajectory evaluation looks at the steps, not just the final answer. I define required checkpoints and forbidden actions rather than one exact path, because many correct trajectories exist. Tool arguments and side effects can be checked deterministically; qualitative qualities like justification need a calibrated judge. I also run multiple trials, since the same task can take different paths on different runs.
Use this as a model, not a script. Saying it in your own words is what makes it survive follow-up pressure.
Related questions
Reading is step one
Now say it out loud.
Understanding an answer and delivering it under pressure are different skills. Rehearse this topic in a live LLM Evals mock interview and get scored feedback on depth, clarity and confidence.