Key facts.
- ITBench found agents resolved only about 11.4% of site-reliability engineering scenarios across real-world IT tasks. source
- On the OpenRCA benchmark, frontier models reach perfect root-cause-analysis accuracy only in roughly the 4 to 12% range. source
- An agent that acts on a wrong diagnosis can remediate the wrong component, leaving the incident active and sometimes causing new harm. source
Why is root-cause analysis where agents fail?
Because diagnosing a real incident requires correctly connecting symptoms across a complex system to the actual cause and the benchmarks show agents do this poorly. ITBench's roughly 11% resolution on site-reliability scenarios and OpenRCA's 4 to 12% perfect-RCA accuracy are not edge cases, they are the central finding: when a production system breaks, the agent is asked to determine why from logs, metrics, traces and topology and it gets the actual root cause right only a small fraction of the time. The danger is what happens next. The agent does not hesitate at its low accuracy; it confidently names a cause and, if allowed, remediates it, restarting a service, scaling a component, changing a config, based on a diagnosis that is wrong roughly nine times out of ten. So the real incident continues because the real cause was not addressed and the agent's remediation may introduce a new problem on top of the original one, because it acted on the wrong system. An incident that needed a correct diagnosis got a confident wrong one and an action that did not help and might have hurt, which is worse than no agent at all.
The confidence is the trap. A low-accuracy diagnosis delivered tentatively could be treated as a hypothesis; delivered confidently and wired to remediation, it becomes an action on the wrong cause and the agent has no signal that its diagnosis is in the 88% that are wrong.
What makes IT ops agents safe?
Separating diagnosis from action and gating remediation. Use the agent to gather evidence and propose a diagnosis, but treat that diagnosis as a hypothesis to verify, not a conclusion to act on, given the 4 to 12% accuracy. Verify the proposed root cause against the evidence before any remediation and require human approval for consequential actions, so a wrong diagnosis is caught before it drives a wrong fix. Where the agent's confidence is high but its accuracy is known to be low, the verification and approval are what stand between assistance and a remediation that makes the incident worse. The agent can accelerate evidence-gathering; the diagnosis and the action need a check the benchmarks say the agent cannot provide for itself.
| Agent role | Outcome at ~4-12% RCA accuracy |
|---|---|
| Diagnose and auto-remediate | Wrong fix, incident continues, new harm |
| Propose, verify, gate remediation | Wrong diagnosis caught before action |
ITBench agents resolve ~11% of SRE scenarios and frontier models hit RCA at 4% to 12%; a more capable one acts on a wrong diagnosis convincingly. (arXiv:2502.05352)
Verifying diagnosis before action is part of what VibeModel does as the Pattern Intelligence Layer. We model the patterns that distinguish a sound root-cause diagnosis from a confident wrong one and gate remediation accordingly, so an IT ops agent helps resolve incidents instead of remediating the wrong thing while the real one continues.
Frequently asked questions
Why not let the agent auto-remediate?
Because it gets the root cause right only ~4-12% of the time. Auto-remediation on a wrong diagnosis fixes the wrong thing and can cause new harm.
Is the agent useless for incidents?
No. It is useful for gathering evidence and proposing hypotheses. The diagnosis and the action need verification and approval.
Why is confident diagnosis dangerous?
Because the confidence does not track the low accuracy, so a wrong diagnosis is delivered as a conclusion and, if wired to remediation, becomes a wrong action.