Key facts.
- MAST analyzed 200+ annotated traces across seven popular multi-agent frameworks and identified fourteen recurring failure modes, with inter-annotator agreement (Cohen's Kappa) of 0.88, evidence that agent failures are specific and categorizable, not a single fog. source
- WildToolBench shows the inconsistency a trace is built to catch: across 57 models on real-world tool-use scenarios, no model exceeded about 15% session accuracy, so a one-off success rarely repeats. source
- A 2025 enterprise survey found most teams running agents in production still cannot trace failures cleanly through multi-step workflows, the gap that keeps "the model hallucinated" as the default explanation. source
- OpenTelemetry GenAI semantic conventions give a standard span shape for capturing prompts, tool calls, and results, so a reasoning trace is an engineering choice, not a research project. source
What changes when the trace exists?
"It hallucinated" is no root cause; a stronger model won't spare you, as MAST found 41-87% of runs failing across fourteen modes. (arXiv:2503.13657)
Picture the same incident twice. An agent confirms a refund that the payment system never processed. In the un-traced version, support escalates, an engineer reads the final output, sees nothing obviously wrong, and files it under "model error." It recurs next week. In the traced version, the engineer opens the run, sees that the agent read a tool response of "refund pending" and treated it as "refund complete," and writes a one-line fix that maps the pending state to a wait-and-recheck. Same model, same incident, completely different ending. The trace did not make the agent smarter. It made the failure legible.
This is why MAST matters beyond its headline numbers. It proves that when you actually annotate traces, failures resolve into a small set of named, recurring modes. That is the opposite of an unknowable black box. A failure with a name has a fix. A failure described only as "hallucination" does not.
Step 4 tool: payments.get_status(order=A19)
result: {"state":"pending"}
Step 5 agent: "Refund complete. Anything else?" <-- defect: pending read as complete
Fix: treat state!="settled" as not-done; recheck or escalate.
Why is "the model hallucinated" so expensive?
It is expensive because it is unfalsifiable and unactionable. It blames the component you cannot directly change and points at no specific behavior, so the same incident returns. Teams that capture reasoning traces break that loop: the second time a failure mode appears, they recognize it, because they named it the first time. The cost of one good trace is a logging layer. The cost of skipping it is paying for the same incident over and over.
The Pattern Intelligence Layer treats the trace as part of the running pattern, not an afterthought bolted on during the outage. The reasoning, the tool calls, and the downstream confirmations are captured on every run, so when something fails you already have the witness. Reliability at the pattern level means you debug the behavior, not the model, and the fix you ship survives the next model upgrade.
Frequently asked questions
Is a normal log not enough?
A log tells you what the agent said. A reasoning trace tells you why it decided, including the tool result it read. The "why" is what turns an incident into a fix.
Will a better model remove the need for traces?
No. MAST found 41 to 87% of runs still failing across leading frameworks, in fourteen distinct ways. You still have to see which one happened.
How much trace is enough?
Enough to replay the decision: the prompt, each tool call and its result, and the step where behavior diverged from intent.