Key facts.
- A 2025 survey of teams with agents in production found observability and evaluation the lowest-maturity part of the stack, and a large majority planning to invest there next, evidence the gap is real and widely felt. source
- MAKER demonstrates that error compounding, not single-step weakness, is what breaks long agent tasks, so a mature team's job is to catch and correct the failing step, which is achievable with annotated traces. source
- WebArena shows production-style failure is frequent even for strong models, with the best GPT-4 agent completing only 14.4% of real web tasks versus 78.2% for humans, so every team will have incidents, and only the observable ones get fixed. source
- The CSA and Google Cloud State of AI Security and Governance survey found only about 26% of organizations have comprehensive AI security governance, and that governance maturity is the strongest predictor of AI readiness, evidence that the teams which ship are the ones that built the visibility to see where runs go wrong. source
- The shipping team reconstructs any run; the stalled team guesses on the same model. A frontier model won't climb for you: MAKER shows 99% accuracy still fails. (arXiv:2511.09030)
What do the maturity levels actually look like?
Level zero is watching outputs. The team sees what the agent said and infers the rest, so every incident is a guess and the postmortem ends in "the model did something." Level one adds per-run traces, so an incident can be reconstructed and named. Level two adds failure classification across runs, so a new incident is recognized instantly as a known mode. Level three adds predictive health, so the team is warned before success rate slides into an incident. The successful deployments live at level two or three. The stalled ones are stuck at level zero, often with a strong model that cannot save them.
The reason the ladder predicts the outcome is response speed. At level zero, mean time to understand a failure is measured in days, because there is nothing to read. At level two, it is minutes, because the failure has a name and a trace. Leadership experiences that difference directly: one team explains and closes incidents, the other accumulates mysteries until confidence runs out.
Can a stronger model substitute for climbing the ladder?
No, and that is the most common expensive mistake. A better model lowers the failure rate somewhat, but WebArena and MAKER both show it never reaches zero and stays inconsistent. So the failures keep coming, and the team that cannot see them keeps guessing, regardless of model. Spending the budget on the next model upgrade instead of on observability buys a slightly lower incident rate and the same inability to explain the incidents that remain. The teams that ship spend on the ladder.
| Maturity rung | Mean time to understand | Typical outcome |
|---|---|---|
| Output-watching | Days, or never | Stalled after incidents |
| Per-run tracing | Hours | Operable |
| Failure classification | Minutes | Ships and sustains |
| Predictive health | Before the incident | Ships and scales |
The Pattern Intelligence Layer is how a team climbs the ladder without rebuilding for every model. Tracing, classification, and predictive health are properties of the pattern, so the maturity you earned does not reset when you swap the model next quarter. Reliability at the pattern level is what keeps a deployment in the group that ships.
Frequently asked questions
We are at output-watching. What is the first move?
Add a reconstructable trace per run. It takes you from guessing to being able to name a failure, which is the biggest single jump on the ladder.
Is predictive health worth it for a small deployment?
Not at first. Get to failure classification before predictive health; the early wins are in being able to explain incidents, not forecast them.
Does a strong model let us skip rungs?
No. It lowers the rate, not the variety, of failures. You still need to see which one hit you.