AI Engineering · LLM applications that ship
How do you manage context in a long-running application?
Short answer
Treat context as a budget: keep stable instructions, retrieve only relevant history, summarise older turns, and evict low-value content before the window fills.
Long conversations and document-heavy tasks eventually exceed the window. Pasting everything is expensive and often lowers quality because the relevant evidence gets buried. Context management decides what the model sees on each turn.
- Keep the system prompt and current task stable; compress or drop older conversation turns.
- Retrieve history by relevance instead of replaying the whole transcript.
- Summarise older turns into durable facts and decisions, and keep the raw text in storage for audit.
- Reserve output tokens in the budget and fail gracefully when the request is too large.
Follow-ups you should expect
Interviewers use these to check whether the first answer was understanding or memorisation.
- 1How would you decide what to summarise versus retrieve verbatim?
- 2What would you measure to know context management is working?
- 3How does this differ for an agent that runs many steps?
Say it in the interview
I treat the context window as a budget rather than a container. The system prompt stays fixed, recent turns stay verbatim, and older turns get summarised into decisions and facts. For long documents I retrieve the relevant sections instead of pasting the whole thing. I also reserve room for the output and have a fallback when the request will not fit, because silently truncating context is how answers become confidently wrong.
Use this as a model, not a script. Saying it in your own words is what makes it survive follow-up pressure.
Related questions
Reading is step one
Now say it out loud.
Understanding an answer and delivering it under pressure are different skills. Rehearse this topic in a live AI Engineering mock interview and get scored feedback on depth, clarity and confidence.