AI Engineering · LLM applications that ship
What should you trace in an LLM application?
Short answer
Trace the full request: prompt version, model, retrieval results and scores, tool calls, token usage, latency per step, output validation and the final answer. Link traces to user feedback.
Traditional logs show that a request happened. LLM debugging needs the inputs and intermediate decisions: which chunks were retrieved, which tools were called, what the model was told, and where time and tokens went. Without that, a bad answer cannot be reproduced or attributed.
- Record prompt and model version, sampling parameters and token counts.
- Store retrieved chunk IDs and scores, not just the packed context.
- Trace each tool call with arguments, result status and duration.
- Link the trace to thumbs-down feedback and eval cases.
- Redact or hash sensitive content according to your data policy.
Follow-ups you should expect
Interviewers use these to check whether the first answer was understanding or memorisation.
- 1Which field would you look at first for a latency regression?
- 2How would you sample traces to control storage cost?
- 3How do traces connect to your eval dataset?
Say it in the interview
I trace the whole request, not just the final answer: prompt version, model, retrieved chunks with scores, tool calls, token usage and step latency. That lets me tell whether a bad answer came from retrieval, the prompt or the model. I also link traces to user feedback so a thumbs-down can become a reproducible eval case. Sensitive content is redacted at write time, because traces otherwise become a compliance problem.
Use this as a model, not a script. Saying it in your own words is what makes it survive follow-up pressure.
Related questions
Reading is step one
Now say it out loud.
Understanding an answer and delivering it under pressure are different skills. Rehearse this topic in a live AI Engineering mock interview and get scored feedback on depth, clarity and confidence.