Key facts.
- METR's randomized controlled trial: 16 experienced developers across 246 tasks were 19% slower with AI tools while estimating they were about 20% faster. source
- The gap between perceived and actual productivity is the core economic risk: the cost is invisible because the work feels effective. source
- This is the economics behind a stark adoption number: MIT's NANDA report found 95% of enterprise GenAI pilots produced no measurable P&L impact, much of that spend going to motion that felt productive without advancing the business goal. source
Why is the felt-productive failure the costly one?
METR's trial found devs 19% slower with AI yet feeling faster; a better model keeps the illusion, false progress the cost. (arXiv:2507.09089)
Because it never triggers the response a visible failure would. When a plan crashes, someone notices and intervenes. When a plan runs, produces output and feels like progress, the cost accumulates with no alarm and the METR result shows how large that gap can be. Skilled developers, measured carefully, were slower with the tool than without it, while sincerely believing the opposite. Translate that to an agent and the danger is clear: a planning approach that feels efficient can be quietly wasting time, tokens and opportunity and because it feels efficient, it survives. The budget overrun that cancels the project starts here, in the difference between how productive the agent feels and how productive it is.
The wasted compute is the visible slice. The wasted opportunity is larger, the better process the agent displaced, the human time spent supervising motion that did not advance the goal, the slower path chosen because it looked faster. None of it shows up unless you measure the agent against the real baseline.
How do you make the cost visible?
Measure cost per outcome against the process the agent replaced, not against an imagined ideal. Count the retries, the re-plans, the human supervision and the compute and compare them to what the old way actually cost. Track whether the agent's felt speed matches its measured speed, because the METR gap says it often will not. A planning approach earns its place when it beats the real baseline on real cost, not when it feels modern. Anything else is paying for the illusion.
| Measurement | What it reveals |
|---|---|
| Felt productivity | Often overstates the gain |
| Cost per outcome vs baseline | The real economics of the plan |
Measuring that honestly is part of what VibeModel does as the Pattern Intelligence Layer. We model the patterns of a planning approach that costs more than it returns, so you catch the felt-productive failure before it quietly outspends the process it was meant to beat.
Frequently asked questions
Does the METR result mean AI tools are useless?
No. It means perceived gains are unreliable and you must measure actual cost per outcome, because the felt speed misled even experts.
What is the biggest hidden cost?
Wasted opportunity and supervision, not raw compute. The motion that feels like progress but does not advance the goal.
How do I set the baseline?
Use the real cost of the process the agent replaces, measured, not the zero-cost ideal you wish you had.