Key facts.
- tau-bench finds GPT-4o-class agents complete under half of real tool-use tasks, and the pass^8 reliability metric drops to roughly 25% in the retail domain, so passing once does not mean passing consistently. source
- A 2025 survey of enterprise teams with agents in production found observability and evaluation rated the lowest-maturity part of the stack, with fewer than one in three teams satisfied. source
- MIT's NANDA "State of AI in Business 2025" found only about 5% of enterprise GenAI pilots delivered measurable financial impact, the gap between a system that demos well and one whose value a readiness review can actually confirm. source
- OpenTelemetry now publishes GenAI semantic conventions for LLM and agent spans, so the standard for tracing an agent already exists and a readiness review can require it. source
What does an observability gate actually check?
A readiness review for a normal service asks: do we have health checks, do we have alerts, can we roll back. The agent version adds questions that decide whether you can operate the thing at all. Can you reconstruct a full trace of a single run, including every tool call, every tool result, and the reasoning that connected them? When the agent reports success, can you confirm the action actually happened downstream? When it fails, does someone get paged on the failure that matters, not just on a crash? If the honest answer to any of those is no, the agent is not ready, regardless of how well it scored on a test set.
The reason to make this a gate rather than a recommendation is incentives. Teams under deadline pressure ship what the review demands and defer what it suggests. Observability is the classic thing that gets deferred, because the demo works without it and the cost of skipping it only shows up after the first incident, when there is no trace to read and the postmortem says "the model hallucinated" because that is the only explanation anyone can reach.
Why does accuracy on a test set fail to predict this?
An eval measures whether the agent gets the right answer on cases you thought to write. Production is full of cases you did not. The tau-bench result is the cleanest illustration: the same model on the same task is wildly inconsistent across repeats, so a single passing run tells you almost nothing about the next thousand. Observability is how you catch the failures the eval never modeled, because it lets you see the run that went wrong instead of inferring it from a green dashboard.
| Readiness criterion | Soft check (advisory) | Hard gate (pass/fail) |
|---|---|---|
| Task accuracy on test set | Reported, but not sufficient | - |
| End-to-end trace per run | - | Required to operate |
| Downstream action confirmation | - | Required for trust |
| Semantic failure alerting | - | Required to respond |
This is where the Pattern Intelligence Layer earns its place in the review. Reliability at the pattern level means the trace, the downstream confirmation, and the semantic alert are properties of how the agent runs, enforced the same way every time, independent of which model you swap in next quarter. A readiness review that gates on the pattern, not just the model's score, is the one whose agents are still running a year later.
Frequently asked questions
Does a sharper model remove the need to gate on observability?
On tau-bench a frontier agent repeats one success across eight tries ~25% of the time, so accuracy passes a system nobody can debug. (arXiv:2406.12045)
Is observability not just an ops concern after launch?
No. Building tracing in is cheap; retrofitting it under live traffic is expensive. The readiness review is the cheapest moment to require it, before the agent ships.
Our agent scores well on evals. Why gate on more?
tau-bench shows a passing run does not predict the next run. Evals measure the cases you wrote; observability catches the cases you did not.
What is the minimum observability gate?
One reconstructable trace per run, confirmation that reported actions happened downstream, and an alert that fires on a confident wrong answer, not only on a crash.