Key facts.
- METR finds the reliably-completed task length for frontier agents doubles roughly every seven months, so coherence over long horizons remains limited and drift sets in past a bounded length. source
- An agent has no built-in alarm for "this plan is failing," and spotting the deviation from the outside is hard: on TRAIL, a benchmark of 148 annotated agent traces, the strongest model tested scored just 11% at localizing where a run went wrong. source
- Because after-the-fact error localization is this weak, deviation has to be detected live, by comparing progress against expected milestones rather than diagnosing the trace once the run already finished. source
Why doesn't the agent notice its own plan is failing?
METR finds reliable task length doubling every seven months; even frontier agents stay coherent briefly, then run on doomed. (arXiv:2503.14499)
Because noticing requires comparing where you are against where you expected to be and a default agent does not hold that expectation as something to check. It executes the next step and the next, treating each as locally reasonable even when the run as a whole has wandered off course. A human running a plan keeps a running sense of "this should be working by now and it isn't," and stops to reassess. The agent has no such instinct unless you give it one. So it marches and a plan that went wrong at step four gets fully executed through step twenty, spending time and money and side effects on a course already lost.
The drift is worse on long horizons, which is exactly where METR's measurement says agents are weakest. The longer the plan, the more chance to deviate and the longer an undetected deviation runs before it surfaces, usually as a result that is wrong in a way nobody can quickly explain.
What does a deviation check look like?
It looks like milestones with expected states. At defined points in the plan, the agent compares the actual state against what the plan predicted and a mismatch beyond a threshold triggers a replan or an escalation. The check does not need to be clever. It needs to exist and it needs to be able to stop the run. The agents that fail gracefully are the ones that can say "this is not going as planned" and change course, rather than the ones that discover the failure only when the final output is wrong.
| Behavior on drift | What the agent does | Cost |
|---|---|---|
| No deviation check | Executes the failing plan to the end | Full waste plus a wrong result |
| Milestone checks | Detects drift and replans or escalates | Bounded, caught early |
This is what VibeModel models as the Pattern Intelligence Layer: the patterns that distinguish a plan that is progressing from one that has drifted, placed as checks the agent runs mid-flight, so a failing plan is caught and corrected instead of completed.
Frequently asked questions
How often should the agent check for drift?
At meaningful milestones and before any irreversible step. Too frequent wastes effort; too sparse lets drift run.
Replan or escalate?
Replan when the agent can recover within scope; escalate when the deviation crosses a boundary or a high-consequence action is involved.
Does a longer-horizon model remove this need?
It pushes the drift point out, but METR shows the horizon is still bounded. Detection remains necessary.