Key facts.
- Finance Agent Benchmark: the best model (o3) reached only ~46.8% accuracy on real-world financial research tasks, at ~$3.79 per query, across 537 expert-authored questions. source
- At under 50% accuracy on real financial work, an agent's claim of completion is not evidence the action succeeded. source
- The gap between a reported action and a verified outcome is where undetected financial errors live. source
Why is the agent's report not enough?
On the Finance Agent Benchmark the best model hits ~46.8%, wrong over half the time; a newer model cannot be trusted the money moved. (arXiv:2508.00828)
Because the report describes what the agent attempted, not what actually happened in the financial systems and those can diverge badly. The Finance Agent Benchmark shows even the best model is right under half the time on real financial tasks, which means a confident report of "payment sent" or "books balanced" comes from a system that is frequently wrong about financial reality. The agent ran the steps it believed would move the money, file the report or reconcile the account and concluded it was done, but completing the steps is not the same as achieving the outcome. The payment may have failed at the rail, the report may contain a wrong figure, the reconciliation may not actually balance and none of that is visible in the agent's account of its own work. In finance, the cost of a falsely-reported success is direct: money that did not move, a report filed wrong, books that do not balance, each of which is a real financial and sometimes regulatory problem.
This is why verification is non-negotiable in finance specifically. The benchmark accuracy says the agent is unreliable enough that its claims must be checked and the domain says the cost of an unchecked false claim is money and compliance, not just inconvenience.
What does verifying a financial action take?
Confirming the outcome in the authoritative system, not the agent. Did the payment actually clear at the rail, does the ledger actually balance, does the filed report match the source figures. The agent proposes and attempts the action; an independent check against the system of record confirms it happened correctly before it is accepted as done. Given the benchmark accuracy, this check is the load-bearing control, because the agent's confidence is not correlated tightly enough with reality to skip it. Reported and real diverge often enough in finance that only verification closes the gap.
| What completion means | Reliability |
|---|---|
| The agent reported it done | Agent is ~47% accurate on real tasks |
| The outcome verified in the system of record | The action actually happened |
Verifying financial outcomes against the system of record is part of what VibeModel does as the Pattern Intelligence Layer. We model the patterns of a genuinely-completed financial action and check the real outcome, so "done" means the money moved and the books balance, not that the agent finished its steps.
Frequently asked questions
Why verify if the agent is confident?
Because it is under 50% accurate on real financial work. Confidence is not correlated tightly enough with reality to trust the claim.
What do I verify against?
The authoritative system of record: the payment rail, the ledger, the source figures. Not the agent's own report.
Isn't this slow?
It is far cheaper than money that did not move or a misfiled report. The verification is the control that makes the agent usable.