Key facts.
- MIT NANDA's State of AI in Business 2025 found roughly 95% of enterprise generative-AI initiatives showed no measurable P&L return, the destination of a pilot that proved capability but not economics. source
- METR's time-horizon work shows the task length frontier models complete reliably is still measured in tens of minutes and rising only slowly, so the long, hard, retry-heavy runs that drive production cost are exactly what a low-volume clean-input pilot never exercises. source
- Agent cost can scale super-linearly as conversations and context grow, so a per-task cost measured small can understate the production bill. source
Why does a capability pilot miss the economics?
Because the things that make an agent expensive in production are the things a pilot is usually built to avoid. A pilot runs on a curated set of inputs, so the hard cases that trigger retries and long reasoning are underrepresented. It runs at low volume, so the costs that only appear at scale, including the long-tail requests and the loops, never accumulate. And it runs for a short window, so the variance in per-task cost, which is real because the agent is probabilistic, is not visible. The agent looks cheap because the pilot was designed to make it look clean and clean is exactly the condition production does not provide. The capability is proven and the cost is not and the cost is what the budget cares about.
A more capable model does not rescue a pilot built this way. It might lower the average cost per task, but the production surprise is in the shape of the workload, the volume and the edge cases, not in the model's per-task efficiency. The quadratic-cost effect is the clearest example: as context and conversation length grow, cost can climb faster than linearly and a short pilot never reaches the length where that bites. You find it in production or you build the pilot to find it first.
How do you build a pilot that models the economics?
Feed it the real input distribution, including the hard cases, so the retry and long-reasoning costs are in the sample. Run it at or projected to, production volume, so the costs that only appear at scale are estimated rather than ignored. Measure cost per correct outcome, not per call, so the error rate and its rework are in the number. And run it long enough to see the variance, because a probabilistic cost is a distribution and a single clean pass is not the budget you will pay. A pilot built this way is less flattering and far more useful: it tells you the production economics before you commit to them, which is the whole point of running a pilot at all.
| Pilot dimension | Capability demo | Economics pilot |
|---|---|---|
| Inputs | Curated, clean | Real distribution, hard cases in |
| Volume | Low | Production or projected |
| Metric | Does it work | Cost per correct outcome |
| Duration | Short | Long enough for variance |
The Pattern Intelligence Layer is where the pilot's economics carry into production. Cost per outcome, retry rate and context growth are tracked at the pattern level from the pilot onward, so the production bill is modeled on real behavior instead of estimated from a clean demo. Reliability at the pattern level is what keeps the scale-up from becoming the moment the economics finally show up.
Frequently asked questions
Will a stronger model keep the pilot economics intact at scale?
A clean low-volume pilot proves the agent works, not what it costs once volume and hard cases arrive; a stronger model does not close that, the surprise lives in cost. (arXiv:2503.14499)
Why did my cheap pilot become an expensive production system?
The pilot likely ran on clean inputs at low volume. Production adds hard cases, retries, long context and scale, which are the cost drivers a capability demo is built to avoid.
What makes a pilot model the economics?
The real input distribution including hard cases, production-level volume, cost-per-correct-outcome as the metric and a long enough run to see the cost variance.
Does a better model remove the need for this?
No. A better model can lower average cost, but the production surprise is in workload shape and scale, like the quadratic context effect, which a short clean pilot still misses.