Key facts.
- On BFCL's multi-turn benchmark, a trajectory is scored correct only if it passes state-based and response-based checks across every turn, and GPT-4o scores about 47.62% in the V3 results, well under 100%.source
- "Measuring Agents in Production" reports that about 74% of deployed agents rely primarily on human-in-the-loop evaluation, with model-based evaluation used mostly as a complement, not a replacement.source
- The same study constrains agents to short, controllable runs (68% to at most 10 steps before human intervention), which is the practical admission that unattended accuracy isn't yet high enough to skip the human.source
Why doesn't high volume guarantee good economics?
Because the economics depend on the net saving per task, and oversight is a cost per task that scales with the volume just like the work does. The model call might be a fraction of a cent, but if a meaningful share of outputs are wrong, you need a human to review them, and that review costs real money per item. When the value of each task is low, there isn't much saving to begin with, so even a modest oversight cost can turn the net positive into a net negative. The agent processed a thousand items cheaply, but a human had to check a large fraction of them, and the human time, not the tokens, set the cost.
The frontier doesn't rescue this. The BFCL multi-turn number is a current, strong model getting about half of multi-turn tool tasks right under a strict full-trajectory scoring rule. A bigger model raises that number somewhat, but not to the point where you can skip the human on a task where a wrong output is unacceptable. that's why production teams default to human evaluation: the unattended accuracy isn't yet high enough to remove it, and removing it's what the cheap-automation pitch quietly assumed.
When does automating the boring task actually pay?
It pays when one of two things is true: the unattended accuracy is high enough that oversight is light or sampled rather than per-item, or the value per task is high enough to absorb the oversight cost and still come out ahead. A narrow, well-scoped task where the agent is reliably right needs only spot checks, so the oversight cost is small and the volume saving survives. A task where being wrong is cheap can tolerate a higher error rate without per-item review. The trap is the combination this article names: high volume, low value, and an error rate that forces per-item oversight. that's where the math fails, and recognizing it before you build is cheaper than discovering it on the invoice.
| Task profile | Oversight needed | Economics |
|---|---|---|
| High accuracy, any value | Spot checks | Saving survives |
| Low accuracy, high value | Per item | Can still pay |
| Low accuracy, low value | Per item | Often net negative |
| Cheap-to-be-wrong | Sampled | Tolerant of errors |
The Pattern Intelligence Layer is where this call gets made on numbers, not hope. Unattended accuracy and the resulting oversight load are tracked at the pattern level for each task, so you can see whether a high-volume task clears the bar for light oversight or falls into the low-value, per-item-review trap. Reliability at the pattern level is what tells you, before you scale, whether automating the boring task actually pays or just moves the cost to a human reviewer.
Frequently asked questions
Does a stronger model on the task remove the oversight?
A more capable agent still clears only about half of multi-turn tool tasks, so the oversight stays, and on low-value work it can erase the saving. (arXiv:2512.04123)
Isn't a cheap model call always worth it at scale?
Only if the outputs are reliable enough to skip per-item review. If a frontier model gets about half of multi-turn tool tasks right (BFCL), the human oversight that catches errors can cost more than the model saved.
Why do production teams keep humans in the loop?
Because unattended accuracy isn't high enough to remove them. A study of deployments found 74% rely primarily on human evaluation, a direct read on how much oversight current agents still need.
When should I not automate a high-volume task?
When its value per task is low and its error rate forces per-item human review. That combination usually makes the oversight cost exceed the automation saving.