Key facts.
- NoLiMa finds long-context performance collapses for models that claim 128K support, with GPT-4o dropping from 99.3% to 69.7% at 32K and most models below half their short-context baseline. source
- An agent asked after the fact why it planned a certain way will reconstruct a plausible reason, which is not the same as a recorded one. source
- Reconstructing the rationale from a raw trace is hard even with the trace in hand: TRAIL, which annotates agent runs as OpenTelemetry spans, found the strongest model localized the actual errors at just 11% accuracy, so the planning rationale has to be captured explicitly rather than recovered later. source
Why isn't the list of actions enough?
Because the actions tell you what happened, not why and the why is where the bug lives. When an agent's plan goes wrong, the actions look locally sensible and the mistake is in the reasoning that connected them, the goal it inferred, the constraint it dropped, the branch it chose. If you logged only the actions, your postmortem is archaeology, reconstructing intent from footprints. If you logged the planning rationale, you can see the exact decision that turned the plan and you can fix it. The same record is what you hand a regulator or an auditor who asks why the agent did what it did, which is a question that comes up first in anything high stakes.
The reason you cannot rely on the agent to explain itself later is that its grasp of the plan decays, which is what the NoLiMa result makes concrete. Over a long run, the reasoning that mattered gets buried and an after-the-fact explanation is a fresh guess, not a memory. The audit trail has to be captured as the plan happens, because it cannot be faithfully recovered afterward.
What do you capture?
The goal the agent was working toward, the sub-plans it considered, the choice it made and the inputs that drove it and the checks it passed or failed. Enough that someone can replay the reasoning, not just the outcome. This is the difference between an agent you can improve and one you can only restart and between a plan you can defend and one you can only apologize for.
| What you record | Can you debug it? | Can you defend it? |
|---|---|---|
| Actions only | By guesswork | No |
| Planning rationale | By replay | Yes |
Capturing that rationale as structured, reusable signal is part of what VibeModel does as the Pattern Intelligence Layer. We model the patterns of a plan worth auditing and record the reasoning behind each decision, so you can debug the agent that drifted and defend the one that did its job.
Frequently asked questions
Can't I just ask the agent to explain afterward?
Its grasp of a long plan decays, as NoLiMa shows. The after-the-fact explanation is a guess. Capture the rationale live.
Isn't logging rationale expensive?
It is cheaper than an un-debuggable incident or an indefensible decision. Capture the reasoning at decision points, not every token.
Who needs the planning audit trail?
Your engineers for debugging and your compliance and legal teams for defensibility. The same record serves both.