Key facts.
- MIT's Project NANDA report "The GenAI Divide: State of AI in Business 2025" found roughly 95% of organizations saw zero measurable return on generative-AI investment, and stated the divide "does not seem to be driven by model quality or regulation, but seems to be determined by approach." source
- "Measuring Agents in Production" surveyed 306 practitioners across 26 domains and found reliability is the top development challenge, with 68% of production agents kept to at most 10 steps before human intervention, a sign teams constrain scope to keep the economics and reliability manageable. source
- The same study found production agents lean on simple, controllable approaches and human evaluation, which is the opposite of the open-ended autonomy a flashy pilot suggests, because open-ended autonomy is what gets expensive and unreliable at volume. source
Why does a pilot pass while production fails?
NANDA found ~95% of GenAI efforts with no return, from integration, not the model; a stronger model won't rescue the economics. (arXiv:2512.04123)
A pilot runs on a small, friendly slice of reality. The requests are curated, the volume is low, and an engineer is watching. At that scale, cost is a rounding error and the occasional failure is caught by the human in the room. Production removes all three protections at once. The requests are messy and long-tailed, the volume is orders of magnitude higher, and nobody is watching every run. Cost that was invisible at ten requests becomes the line item that sinks the business case at ten thousand, and reliability that looked fine under supervision shows its real failure rate when it runs unattended. The pilot was never an economic test. It was a capability test, and the two are not the same.
The NANDA finding that the divide is about approach, not model quality, is the part that catches teams out. The instinct after a disappointing rollout is to reach for a better model. But if the economics break because volume multiplied a per-request cost or because reliability fell when supervision was removed, a stronger model changes neither. It may even cost more per request. The fix is in the approach: scope, integration, and a cost model that was built for production volume from the start.
What does the economic test actually measure?
It measures cost per successful outcome at real volume, with real failure rates and real oversight cost included. A pilot reports the happy-path cost of one good run. The economic test reports what a thousand runs cost when some fraction fail, retry, escalate, or need a human to clean up. That number is what decides whether the agent survives. The teams in the "Measuring Agents in Production" study who keep agents to short step-counts and lean on human evaluation are not being timid. They are keeping the production cost model inside the bounds where the math works, which is exactly the discipline a pilot does not force you to have.
| Dimension | Pilot | Production at scale |
|---|---|---|
| Request mix | Curated, friendly | Messy, long-tailed |
| Volume | Low | Orders of magnitude higher |
| Supervision | Engineer watching | Mostly unattended |
| Cost that matters | One good run | Per outcome incl. failures + oversight |
The Pattern Intelligence Layer is where the economic test runs before the rollout, not after. Cost per successful outcome, failure rate, and oversight cost are tracked at the pattern level on production-shaped traffic, so the gap between a pretty pilot and a sustainable agent is a number you see in advance. Reliability at the pattern level is what turns "the demo worked" into "the agent pays," which is the only version that survives the volume.
Frequently asked questions
My pilot was a success. Why would production be different?
Because the pilot tested capability on a friendly slice, not economics at volume. Higher volume, messier requests, and no supervision change the cost and reliability math entirely.
Will a better model fix a pilot that didn't scale economically?
Usually not. MIT NANDA found the divide is about approach, not model quality. If volume or reliability broke the economics, a stronger model does not change them and may cost more.
How do I test economics before committing?
Measure cost per successful outcome on production-shaped traffic, with real failure rates and oversight cost included. That number, not the demo, tells you if the agent pays.