LLM Evals · Turn quality into a signal
How is evaluating an agent different from evaluating a chatbot?
Short answer
Agents need outcome and trajectory evaluation: did they achieve the goal, and did they take safe, efficient, correct steps with valid tool calls along the way.
A chatbot's output is the answer. An agent's output includes a sequence of decisions and actions. A good final answer reached through destructive or wasteful steps is still a failure. Evaluation has to inspect both the result and the path.
- Outcome: did the task succeed end to end?
- Trajectory: were the steps valid, efficient and safe?
- Tool use: correct tools, valid arguments, handled errors.
- Recovery: did the agent notice failure and adapt?
- Cost and latency: how many steps and how much spend per success?
Follow-ups you should expect
Interviewers use these to check whether the first answer was understanding or memorisation.
- 1How would you score a successful task that took twenty unnecessary steps?
- 2What safety checks belong in an agent eval?
- 3How many trials per task would you run?
Say it in the interview
A chatbot is evaluated on its answer; an agent also has to be judged on how it got there. I check the outcome, the trajectory, tool arguments, error recovery and cost per success. A correct answer reached by deleting the wrong data is still a failure. Because agents vary across runs, I evaluate multiple trials per task and report a success rate rather than one lucky trajectory.
Use this as a model, not a script. Saying it in your own words is what makes it survive follow-up pressure.
Related questions
Reading is step one
Now say it out loud.
Understanding an answer and delivering it under pressure are different skills. Rehearse this topic in a live LLM Evals mock interview and get scored feedback on depth, clarity and confidence.