Context management is an architecture decision, not a prompt trick

A 128K context window is not a solution to the context management problem. It's a delay. Given a complex enough task — a multi-day workflow, a research process, a code generation session with dozens of iterations — you will hit the limit. The question is whether you've designed for that moment or whether you discover it in production.

The teams building agents that work at real scale don't think about context management as a prompt engineering problem. They design their context architecture the same way they design their data architecture: explicit decisions about what lives where, for how long, at what granularity.

The four layers of context architecture

Layer 1: Working context (in-window)

This is the live conversation: the system prompt, the current user request, recent tool call outputs, and the agent's in-progress reasoning. Everything the model can access directly.

Working context is fast but finite. The design decision here is: what goes in, what gets compressed out, and when. Working context should contain only what's needed for the current reasoning step. Everything else is waste that eats into the budget.

Layer 2: Episodic memory (summarized, retrievable)

When the agent completes a step or crosses a context threshold, compress what happened into a summary. Store the summary in a durable location (database, vector store). The summary is not in context by default — it's retrieved when relevant.

This is the mechanism that lets an agent reason across sessions. "What did we decide in the last session?" retrieves a summary rather than re-reading a full transcript. The summary is compact. It captures decisions, state, and outcomes without the verbose tool output that generated them.

Layer 3: Semantic memory (indexed knowledge)

Facts, domain knowledge, and long-term state that the agent needs access to but doesn't need inline. Stored as embeddings in a vector store, retrieved by semantic similarity when relevant to the current step.

This is where customer profiles, product catalogs, and accumulated knowledge live. The agent retrieves a customer's history when answering a question about that customer — it doesn't load the full history into context for every request.

Layer 4: Structured state (explicit, schema'd)

The agent's model of the current task: what steps have been completed, what the outputs were, what's next. This is not unstructured conversation history. It's a data object with a schema, stored in a database, updated atomically at each checkpoint.

This layer is the source of truth for task state. Context is a view of this state, not the state itself. If context is lost or corrupted, the agent can reconstruct its position from structured state.

The context budget

Before writing any agent code, define your context budget: how many tokens does each component consume, and what's the total budget for a typical task execution?

A rough template for a complex task agent:

If your tool outputs are uncompressed JSON from a 100-record database query, that one tool call blows the budget. The compression step is not optional.

Compression strategies

Field extraction. Before adding tool output to context, extract only the fields the agent actually needs. A user record with 40 fields where the agent needs 4 should add 4 fields' worth of tokens to context, not 40.

def compress_user_record(raw: dict) -> dict:
    return {
        "id": raw["id"],
        "name": raw["name"],
        "subscription_status": raw["subscription"]["status"],
        "last_active": raw["activity"]["last_login"]
    }

Summarization. For long documents, search results, or accumulated conversation, run a summarization step before the content enters the working context. The summary should capture what the agent needs to reason from, not reproduce the full content.

Pagination. For list results, don't retrieve all records at once. Retrieve a page, process it, store the output, retrieve the next page. Each page is processed and discarded. The agent's working context never holds more than one page at a time.

Sliding window. For conversational agents with long sessions, keep only the N most recent turns in context plus a rolling summary of what came before. The summary is updated at each turn.

Context boundaries between agent steps

Long-horizon agents should treat major step transitions as context boundaries. At each boundary:

1. Summarize what was accomplished in this step
2. Write the summary and any outputs to structured state storage
3. Start the next step with a fresh context loaded from the summary and relevant retrieved memory

This is the architecture that makes "resume from checkpoint" possible. Each step starts clean. The agent's knowledge of what came before comes from structured storage, not from an ever-growing conversation history.

When to use which approach

Short tasks (< 10 steps, single session): Simple sliding window + field compression. No need for episodic memory or cross-session state.

Medium tasks (10-50 steps, single session): Periodic summarization at checkpoint boundaries. Structured state for step tracking. Vector retrieval for domain knowledge.

Long tasks (50+ steps, multiple sessions): Full four-layer architecture. Cross-session episodic memory. Structured state as source of truth. Context is assembled fresh at each session start from stored state.

Always: Field compression on tool outputs. Budget tracking. No raw API responses in context.

The context window is a working memory. Working memory is limited. Architects who understand this build the supporting structures — compressed storage, retrieval systems, summarization pipelines — before they need them. Architects who don't understand this hit the context limit in production and patch it with a bigger model.

A bigger model has a bigger window. The underlying design problem is the same.