LLM Evals · Turn quality into a signal
How do you evaluate tool calls?
Short answer
Check tool selection, argument validity, permission compliance, result handling and whether the call was necessary. Deterministic checks cover most of this better than a judge.
Tool calls are structured, which makes them easier to evaluate than prose. You can assert the tool name, validate arguments against a schema, verify permissions and inspect whether the agent used the returned result. The harder question is whether the call was appropriate at all.
- Correct tool for the intent, with no unnecessary calls.
- Arguments satisfy schema and business constraints.
- No calls outside the user's permissions or the task scope.
- Errors are handled and not retried blindly.
- The observation is actually used in the next step.
Follow-ups you should expect
Interviewers use these to check whether the first answer was understanding or memorisation.
- 1How would you test that an agent does not call a destructive tool without approval?
- 2What would you assert about a retried tool call?
- 3When would a judge be better than deterministic assertions?
Say it in the interview
Tool calls are structured, so I check them deterministically: the right tool, valid arguments, permission compliance, proper error handling and whether the result was used. I also test failure modes by mocking timeouts and malformed responses, because happy-path mocks hide the interesting bugs. The judgement call is whether the tool was needed at all, and that is where a calibrated judge can help.
Use this as a model, not a script. Saying it in your own words is what makes it survive follow-up pressure.
Related questions
Reading is step one
Now say it out loud.
Understanding an answer and delivering it under pressure are different skills. Rehearse this topic in a live LLM Evals mock interview and get scored feedback on depth, clarity and confidence.