Key facts.
- RULER found only 4 of 17 long-context models (GPT-4, Command-R, Yi-34B, Mixtral) held satisfactory performance at 32K despite all claiming 32K or more.source
- The drop appears even when models score near-perfect on simple needle-in-a-haystack tests, so the advertised length overstates the usable length.source
- For a long-running agent, the context fills with history and the planning-relevant facts get buried where the model uses them worst.source
Why does planning decay over a long run?
Because the context that the plan depends on degrades in two ways at once. It fills up, pushing the planning-relevant facts further from the edges where the model attends best and it ages. Early decisions and constraints sit deep in a window the model now uses poorly. RULER's result is the uncomfortable version of this: the 32K you paid for is often not 32K of usable reasoning. So an agent ten steps into a long task is planning off a context that nominally contains everything it needs and functionally does not. The plan it produces is coherent and subtly wrong. That happens because it is reasoning over information it can no longer fully access.
Nothing alerts the agent to this. The context did not error. It just stopped working and the agent kept planning as if it were intact. Is how a long-running agent drifts from competent to confidently confused.
How do you keep the context working?
Manage it like a resource with a real limit below the advertised one. Summarize history into compact, high-signal state instead of letting raw transcript accumulate. Prune what the plan no longer needs. Re-ground the agent on the active goal and constraints at intervals. The facts that matter stay where the model uses them well. The agent does not need a bigger window. It needs the planning-relevant context kept small, fresh and near the edges where attention actually works.
| Context strategy | What the agent plans on |
|---|---|
| Trust the big window | Buried, aged facts the model uses poorly |
| Summarize, prune, re-ground | Compact, current, usable state |
Keeping that working state compact and current is part of what VibeModel does as the Pattern Intelligence Layer. We model the patterns of the context a plan actually needs and keep it usable over a long run. The agent's planning holds up at step fifty the way it did at step five.
Frequently asked questions
Won't a longer context window solve this?
RULER shows the usable length trails the advertised one. A bigger window moves the cliff, it does not remove it. Active management does.
What should I summarize?
The goal, the active constraints and the decisions that still bind. Drop the raw chatter that no longer informs the plan.
How often should I re-ground?
Often enough that the planning-relevant facts never sink deep into a long, aged context. Tie it to milestones.