Key facts.
- On SWE-bench Pro, leading models resolve only about 23% of real-world software-engineering issues (reported), a direct warning against assuming the agent beats the baseline on the hard work the status quo already handles. source
- WildToolBench, across 57 models and 1,024 real-world tool-use tasks, found none exceeded about 15% session accuracy, so unclear value is exactly what a missing baseline comparison hides behind a one-off success. source
- Cost analysis shows agent spend is set at runtime by tokens, tool calls and retries and climbs faster than linearly as context grows, so a fair benchmark has to measure cost per correct outcome on the live workload rather than a per-call list price. source
Why is the status quo the comparison that matters?
Because the agent is not replacing nothing, it is replacing whatever you do today and that is the only honest yardstick. A process you already run has a real cost per outcome and a real error rate and those are the numbers the agent has to beat to be worth the switch. Compare the agent to an idealized blank page and it always wins, because the blank page has no cost. Compare it to the actual process and you measure the thing leadership is buying: a better cost per correct outcome on the same work. SWE-bench Pro is the cautionary example because it runs exactly this kind of comparison on real engineering issues and leading models resolve only about a quarter of them, leaving the rest to the people the agent was supposed to replace.
A stronger model does not let you skip this. The SWE-bench Pro models are current and capable and still leave most issues unresolved, which means model quality alone does not guarantee the agent beats the baseline on a specific workload. The only way to know is to run both and measure. Sometimes the agent wins clearly, sometimes it wins on cost but not speed, sometimes it loses and the benchmark is what tells you which, before the spend is committed.
How do you run a fair benchmark?
Hold the workload and the outcome metric identical across both arms. Use the same real inputs, not a cherry-picked sample, because the status quo handles the hard cases too. Count the full cost on each side: for the agent, tokens, tools, retries and the oversight to catch its mistakes; for the status quo, the labor and tooling it already consumes. Score on cost per correct outcome, which folds the error rate into the number, so a cheaper-but-wrong agent does not look like a win. And run it long enough to see the variance, because an agent's cost is a distribution and one lucky pass is not a baseline. Done this way, the benchmark gives leadership a number they can defend and gives you the early read on whether the agent earns its place.
| Benchmark choice | Flattering but useless | Honest |
|---|---|---|
| Comparison | Agent vs nothing | Agent vs current process |
| Metric | Per-call list price | Cost per correct outcome |
| Inputs | Easy sample | Same real workload |
| Duration | One pass | Long enough to see variance |
On SWE-bench Pro, leading models resolve only ~23% of real issues, so the hunch that the agent obviously beats the old process is the one a frontier model did not earn. (arXiv:2509.16941)
The Pattern Intelligence Layer is where this benchmark keeps running after launch. Cost per correct outcome for the agent and the process it replaced are tracked side by side at the pattern level, so the comparison stays live as volume and models change and the moment the agent stops beating the baseline shows up as a trend. Reliability at the pattern level is what keeps the business case honest past the demo.
Frequently asked questions
Why not just compare to the model's list price?
Because list price ignores retries, oversight and the error rate. Cost per correct outcome on the real workload is the number that maps to what the business actually pays.
Does the SWE-bench Pro result mean agents never help?
No. It means the win is not automatic. Leading models resolve only about a quarter of real issues, so the gain depends on the task, which is exactly why you benchmark instead of assume.
What if the agent loses on speed but wins on cost?
That is a real and common outcome and the benchmark surfaces it. You then decide whether the cost saving is worth the speed trade for that workload.