Key facts.
- tau-bench: state-of-the-art agents succeed on under 50% of customer-service tasks and show pass^8 reliability under 25% in retail, evaluated by comparing the final database state to the goal state. source
- The benchmark grades by actual end state, not the agent's claim, which is exactly the verification production support usually omits. source
- A claimed resolution and a real one diverge whenever the agent reports success without confirming the downstream effect. source
Why does the agent claim a resolution that did not happen?
Because it is reporting on its intent, not on the outcome. The agent ran the steps it believed would resolve the issue and concluded it was done, but completing the steps is not the same as achieving the result and tau-bench measures the gap precisely by checking the actual end state rather than the agent's say-so. Under that honest grading, agents succeed less than half the time and, more tellingly, repeat a success across eight tries under a quarter of the time in retail. So the agent that closed this ticket may have genuinely resolved it or may have run plausible steps that did not land and from its own report you cannot tell which. It says resolved because it finished and finishing is what it was optimizing, not confirming.
The cost lands on the customer and on trust. The refund the agent reported was never issued, the fix it claimed did not take and the customer discovers this later, now doubly frustrated because they were told it was handled. The missing step was never the resolution itself. It was confirming the resolution was real before declaring it.
What does verifying a resolution take?
Confirm the outcome in the system of record before you call it resolved. Did the refund actually post, did the ticket status actually change, did the underlying issue actually clear. This is the same standard tau-bench uses to grade, the real end state versus the goal state and it is the standard production support should apply before telling a customer their problem is solved. The agent proposes a resolution; the verification confirms it; only then does the ticket close and the customer get the good news.
| Closing logic | What resolved means |
|---|---|
| Trust the agent's claim | The agent finished its steps |
| Verify the outcome | The result actually happened |
Building that verification is central to what VibeModel does as the Pattern Intelligence Layer. We model the patterns of a genuinely-resolved issue and check the real outcome against them, so resolved means the refund posted and the issue cleared, not that the agent ran out of steps.
Frequently asked questions
Won't a more accurate model verify its own resolutions?
On tau-bench, agents repeat a retail fix under 25% of eight tries; a better model closing a ticket once cannot be trusted it stuck. (arXiv:2406.12045)
Why not trust a high-performing agent's claim?
tau-bench shows even top agents are inconsistent, with pass^8 under 25% in retail. A single claimed success does not mean the action reliably landed.
What should verification check?
The actual system state: refund posted, status changed, issue cleared. Grade by outcome, not by the agent's report.
Does this slow down resolutions?
Slightly and it prevents the far costlier false resolution that brings the customer back angry. The check is cheap insurance.