Key facts.
- MAKER frames the core production cost: per-step errors compound, so a 99% per-step model fails over a long chain, and the recovery (retries, voting, re-attempts) is exactly the cost a clean test run never incurs. source
- tau-bench's pass^k metric shows frontier agents are inconsistent across repeats, with retail reliability under a quarter across eight tries, so production hits the retry-and-recover path far more than a test set suggests. source
- HammerBench found parameter-name errors a primary cause of multi-turn function-calling failure, a concrete example of the kind of error that triggers a costly retry in production but is easy to miss in a short test. source
Why is the test number always too low?
Testing runs curated cases at low volume; production adds failures that compound each step. MAKER shows 99% per-step accuracy still fails a long chain, so even a frontier agent runs recovery (tau-bench). (arXiv:2511.09030)
Because testing selects against the conditions that drive cost. Test cases are curated, so they skew toward the ones the agent handles, which means low failure rates and few retries. Test volume is small, so the long-tail expensive failure may not appear at all. And test tasks are often short, so the error compounding that MAKER describes does not have enough steps to bite. Production reverses all three: real cases include the hard ones, real volume surfaces the rare expensive failure, and real tasks are long enough for per-step errors to compound into retries. The result is a per-task cost in production that is the test number times a failure-and-recovery multiplier the test never measured.
The HammerBench finding is a good illustration of how this hides. A parameter-name error is invisible in a quick test where the inputs are clean, but in production with imperfect user input it becomes a primary failure cause, and each failure is a retry you pay for. The test did not see it because the test did not stress it. Production does both.
How do you estimate the production number before you have it?
Derive a retry-and-recovery factor from your measured first-pass success rate, which tau-bench and HammerBench both suggest is well below one on realistic tasks. Apply it to the clean test cost, then add the long-context overhead for the multi-step tasks that compound. Validate against a production-shaped pilot that includes hard cases and realistic volume, not a curated demo. The number you get is higher than the test figure, and it is the one that lets you decide and budget honestly. The single highest-impact move is raising first-pass success by narrowing scope, because that shrinks the recovery multiplier directly.
| Cost driver | Test environment | Production reality |
|---|---|---|
| Case mix | Curated, easy-skewed | Includes the hard ones |
| Failure rate | Low | From real distribution |
| Retries / recovery | Rare | Constant, per tau-bench |
| Task length | Short | Long enough to compound |
The Pattern Intelligence Layer is where the recovery multiplier becomes a measured property instead of a post-launch discovery. First-pass success, retry factor, and error-recovery cost are tracked at the pattern level, so the production estimate reflects how the agent actually runs under real conditions. Reliability at the pattern level is what closes the gap between the test number and the invoice.
Frequently asked questions
Our test costs are stable and low. Why budget higher?
Tests are curated, low-volume, and short, so they miss the failures, the rare expensive runs, and the error compounding that drive production cost. Apply a retry-and-recovery factor.
What is error compounding in cost terms?
Per-step errors accumulate over a long task, so longer chains fail more and trigger more recovery. MAKER shows even 99% per-step accuracy fails over enough steps.
What cuts the production premium most?
Raising first-pass success by narrowing scope. It shrinks the retry-and-recovery multiplier that turns the clean test number into the production one.