Key facts.
- WebArena measured the best GPT-4-based agent completing 14.41% of realistic web tasks versus 78.24% for humans, a gap that means most agent attempts on such tasks fail and incur retry or escalation cost. source
- tau2-bench evaluates conversational agents in a dual-control setting across airline, retail, and telecom domains, scoring only fully correct task completions, which is the binary outcome a cost-per-outcome model should use as its unit. source
- Industry cost analysis shows agent runs grow more expensive as context accumulates, so the cost of a single attempt is not constant and a model must use realistic, not pilot-length, runs (reported). source
- Cost per run ignores failed runs and cleanup; on WebArena the best agent finishes ~14% versus ~78% for humans, and a bigger model narrows that slowly. (arXiv:2307.13854)
What goes into a cost-per-outcome model?
Four terms. The cost of a successful run. The cost of the failed runs that preceded it, which is the per-run cost divided by the success rate. The cost of human handling for the failures that escalate. And the cost of rework when a wrong outcome ships and has to be corrected downstream. Add those and divide by successful outcomes, and you have the agent's real unit cost. The benchmark gaps tell you the success rate is the term that dominates: at a 14% rate, you pay for roughly seven attempts per success, so the failed-run cost is most of the total, not a footnote.
Then build the same model for the process the agent replaces, whether that is a human, an RPA bot, or a script. A human has a high success rate and an hourly cost. A script has near-perfect reliability on its narrow task and almost no marginal cost. The agent wins only where its cost per outcome, including the failure tax, comes in under the incumbent's. That comparison is frequently closer than the demo suggested, and sometimes it goes the other way.
Why does cost per run flatter the agent so badly?
Because it silently assumes a 100% success rate, and the benchmarks show that assumption is false by a wide margin on realistic tasks. Cost per run treats every attempt as a delivered outcome. Cost per outcome counts only the ones that worked and charges the rest against them. The lower the success rate, the larger the divergence, which is why narrow agents on high-success tasks have honest economics and broad agents on hard tasks often do not. Use the right unit and the agent has to earn the comparison rather than win it by definition.
| Term | Cost per run | Cost per outcome |
|---|---|---|
| Successful run | Counted | Counted |
| Failed runs | Ignored | Charged against successes |
| Human handling | Ignored | Included |
| Downstream rework | Ignored | Included |
The Pattern Intelligence Layer is where cost per outcome becomes a standing measurement rather than a one-time spreadsheet. Success rate, failure tax, and rework are tracked at the pattern level for both the agent and the incumbent, so the comparison is current and defensible. Reliability at the pattern level is what lets you prove the agent is cheaper, on the only unit that matters.
Frequently asked questions
Why not just compare cost per run to a human's hourly rate?
Because cost per run assumes every attempt succeeds. At a realistic success rate, the failed runs and rework dominate the agent's true cost per outcome.
What success rate should I assume?
Measure it on production tasks. Benchmarks like WebArena show it can be far below intuition on open-ended work, so do not inherit the demo's rate.
When does the agent clearly win?
On narrow, high-success tasks where the failure tax is small and the incumbent is expensive. Broad, low-success scopes are where the comparison flips.