Context windows are not memory. They're more like a whiteboard that gets smaller as the conversation goes on.
You start a long-horizon agent task. The system prompt and initial goal occupy the first few thousand tokens. Each tool call adds its output. Each step adds its reasoning. Each document the agent reads adds its full text. By the midpoint of a complex task, the whiteboard is mostly covered with tool outputs and intermediate reasoning, and the original goal — the thing that started all of this — is buried somewhere in the first 5% of the context.
The model at the end of that context is not equally attentive to all of it. Research on long-context models consistently shows that information in the middle of a long sequence is recalled less reliably than information at the beginning or end. The goal that the task depends on may be technically present but functionally invisible.
What this looks like in practice
Goal drift. The agent starts executing on a modified version of the goal. Not because it was instructed to change course, but because the original goal was stated once, at the beginning, and the weight of subsequent context has gradually shifted the agent's working model of what it's trying to do. The final output is coherent with recent context. It misses the original intent.
Premature termination. The agent concludes it has completed the task. But it completed a subset of the task — the parts that were still vivid in recent context. The parts specified earlier, buried in the middle of a long context window, were silently dropped.
Silent truncation. Context limits are hard. When the model reaches its context limit, something has to be cut. Depending on the framework, this might be the oldest messages, the longest tool outputs, or a heuristic selection of what seems least relevant. None of these truncation strategies are correct — each can cut something the agent needs. And the agent typically continues without flagging that it lost information.
Reasoning about outdated state. A tool returned a result at step three. By step twenty, that result has been superseded by a later tool call. But the step-three result is still in context, competing with the step-twenty result. The agent can end up reasoning from the earlier, wrong state.
The math on context pressure
128K context sounds like a lot. But fill it with tool outputs and you run out fast.
A database query returning 100 records at 50 tokens per record: 5,000 tokens. A document the agent reads in full: 3,000-10,000 tokens. Six API responses with metadata: 3,000 tokens. Accumulated reasoning over 20 steps: 4,000 tokens. System prompt and goal: 2,000 tokens.
You're at 17,000-24,000 tokens before the task is halfway complete. With a complex agent doing real work across many tools, 128K fills up in a single session.
What to build before this bites you
Compress tool outputs before they enter context. When a tool returns a large response, don't put the full response in context. Extract the fields the agent actually needs and discard the rest. A database record with 40 fields where the agent needs 4 should contribute 4 fields worth of context, not 40.
Summarize periodically. At defined checkpoints — after every N steps, or when context usage crosses a threshold — summarize what has been accomplished, what the current state is, and what the goal is. Replace accumulated context with the summary. Re-anchor to the original goal explicitly.
Inject the goal at intervals. Don't rely on the model remembering a goal stated thousands of tokens ago. Re-inject the original task specification at each major decision point. This is not redundant — it's compensating for a real attention bias that long contexts create.
Monitor context usage. Build context utilization tracking into your agent framework. When context is 50% full, you should know. When it's 80% full, you should be actively summarizing or truncating deliberately. Don't let the hard limit hit you without warning.
Design for context horizons. Before deploying a complex agent, calculate how many tokens a typical task will require. If it exceeds your context window, redesign the task structure. Break it into sub-tasks, each with a fresh context. Don't discover the context limit in production.
Faultmap tests your agent against tasks of realistic complexity. It measures context growth rates, identifies where goal drift begins, and flags whether your context management strategy can sustain the task to completion. You see the context ceiling before you hit it in production.
The context window is not a limitation to work around. It's a constraint to design for. Every token you put in context is a token that competes with your goal. Budget accordingly.