Key facts.
- ITBench found agents achieve only about an F1 of 0.35 on financial-operations anomaly detection, the skill of catching when something is wrong.source
- Across ITBench domains, agents resolved a small fraction of real operational scenarios, so their reports of success are unreliable.source
- In supply chains, a reported outcome and the physical reality, order placed, inventory counted, payment settled, can diverge with real cost.source
Why does the report diverge from reality?
Because the agent reports on its action. Not on the physical and financial outcome and in a supply chain the gap between the two is where the cost lives. The agent submits an order and reports it placed, but the supplier system rejected it. It reconciles inventory and reports a count, but the count is wrong; it triggers a payment and reports it settled, but it did not clear. The agent's report reflects that it executed the steps. Is not the same as the order shipping, the inventory being accurate or the money moving, and ITBench shows agents are specifically weak at the anomaly-detection skill that would let them notice the discrepancy, managing only an F1 around 0.35 on catching when something is off. So the agent does not flag that its action failed. That happens because detecting that failure is precisely what it is bad at and the false report flows downstream: a stockout because the order that was reported placed never went through, a planning error because the inventory count was wrong, a financial discrepancy because the payment did not settle. The physical and financial reality has to be checked against the report. That happens because the agent's report is not evidence the outcome occurred and the agent cannot reliably tell that it did not.
The physical dimension makes this distinctly hard in supply chains. Verifying that an order shipped or inventory is accurate means checking against the warehouse, the carrier, the supplier, the real-world systems and sometimes the physical goods, not just the agent's internal record. Is why the verification has to reach into the systems of record rather than trusting the agent's account.
What does verifying the outcome take?
Confirmation against the systems of record, not the agent's report. Verify that the order actually landed in the supplier system, that the inventory count reconciles with the warehouse. That the payment cleared in the ledger, before treating any of them as done. Given that agents are weak at detecting anomalies, the verification cannot rely on the agent noticing its own failure. It has to be an external check against the real operational systems. Where the verification fails, the discrepancy is caught and corrected before it cascades into a stockout. An overstock or a financial error. The agent can act; the verification confirms the action produced the physical and financial outcome it reported.
| What completion means | Reliability |
|---|---|
| The agent reported it done | Agent is weak at detecting its own failures |
| The outcome verified in the systems of record | The order shipped, the count reconciles, the payment cleared |
Verifying outcomes against the real systems is part of what VibeModel does as the Pattern Intelligence Layer. We model the patterns of a genuinely-completed supply chain action and check the physical and financial reality. A reported outcome means the order shipped and the books reconcile rather than that the agent ran its steps.
Frequently asked questions
Why not trust the agent's report?
ITBench shows agents are weak at anomaly detection, so they often do not notice their own action failed. The report is not evidence the outcome occurred.
What makes supply chain verification harder?
The outcome is physical and financial, order shipped, inventory accurate, payment cleared, so verification must reach into the real systems of record.
What does an unverified failure cost?
A stockout from an order that never went through, a planning error from a wrong count or a financial discrepancy from a payment that did not settle.