Key facts.
- FinQA shows financial-reasoning accuracy falls to ~22.78% on 3+-step programs as small arithmetic errors cascade through the chain.source
- A cascaded error frequently produces a plausible-looking wrong value not an obviously broken one, so it evades surface checks.source
- An agent has no internal signal that a number does not reconcile, so a plausible discrepancy flows downstream unflagged.source
Why are these discrepancies so hard to detect?
Because they are designed, by the nature of the failure, to look fine. A gross error, a number off by an order of magnitude, gets caught immediately because it is obviously wrong. The dangerous error is the small one: a rounding mistake, a wrong cell, a step in the wrong order. That produces a value close enough to right that it passes a glance and reconciles on the surface. FinQA's multi-step collapse is exactly this mechanism, small errors cascading to a plausible wrong total and the agent reports it with the same confidence as a correct one because it has no sense that the number fails to reconcile against reality. So the discrepancy enters the books looking like every other entry and the only way it surfaces is a deeper reconciliation. An audit or a downstream consequence weeks later, by which time it is expensive to trace and unwind. Some never surface at all, sitting in the records as a permanent small error nobody finds.
The asymmetry is what makes this a priority. The loud error is cheap because it is caught at once. The quiet, plausible error is expensive precisely because it is not and an agent prone to small cascading mistakes is a steady source of exactly the quiet, plausible kind.
What surfaces the quiet discrepancy?
Reconciliation against the source of truth, at each step rather than only at the end. Check that each intermediate figure reconciles with the underlying data. A small error is caught where it happens instead of after it has cascaded into a plausible-looking total. Use deterministic computation for the arithmetic so the cascade does not start and flag any figure that does not reconcile rather than trusting that a reasonable-looking number is correct. The point is to never accept a financial figure on plausibility alone. That happens because the discrepancies that cost you are the ones that look perfectly plausible.
| Checking approach | What it catches |
|---|---|
| Surface check, end-of-process | Loud errors only; plausible ones pass |
| Step-wise reconciliation to source | Small discrepancies before they hide |
FinQA accuracy collapses past three steps and a cascaded error lands plausible; a bigger model's wrong number trips no alarm at all. (arXiv:2109.00122)
Catching the plausible discrepancy is part of what VibeModel does as the Pattern Intelligence Layer. We model the patterns of a figure that does not reconcile and check at each step. The small, quiet error is caught where it happens instead of discovered at audit or never.
Frequently asked questions
Why not just check the final totals?
A cascaded error can produce a plausible final total that reconciles on the surface. Step-wise reconciliation catches it before it hides.
Why are small errors worse than big ones?
Big errors are caught immediately. Small plausible ones pass surface checks and surface late, at audit or never, which makes them expensive.
How do I stop the cascade?
Use deterministic computation for arithmetic and reconcile each intermediate figure against the source, rather than trusting plausible numbers.