Key facts.
- FinQA: GPT-4 reaches roughly 76% exact-match on financial QA overall, but accuracy falls to about 22.78% on programs requiring three or more reasoning steps. source
- FinQA grades by exact match, where the answer must be numerically identical to the gold value, not approximately close, reflecting real accounting standards. source
- The reasoning chain, not retrieval, is the bottleneck: small arithmetic errors at any step cascade into a wrong final number. source
Why does the agent get multi-step finance wrong?
Because financial calculations are chains of arithmetic and a chain is only as accurate as its weakest step. FinQA makes this concrete: the model must find the right numbers in a table and execute the right operations in the right order and while it does this acceptably for one or two steps, the accuracy collapses to roughly a fifth on three-plus-step programs. Each step introduces a chance of a wrong cell, a wrong operation or a rounding error and those compound, so a calculation that needed five correct steps fails most of the time even when each step is individually plausible. In finance, exact match is the standard, not "close enough," because a transaction misclassified or a figure off by a cent is wrong and the agent's tendency to be approximately right is exactly the wrong property for a domain that demands exactness.
Misclassification is the same failure in categorical form. The agent assigns a transaction to a plausible category, which is right when the plausible category is the correct one and wrong otherwise and it does this with the same confidence either way. There is no internal alarm that says "this multi-step total does not reconcile" or "this category does not fit," so the error flows downstream as if it were correct.
What makes the numbers trustworthy?
Deterministic verification, not model confidence. Pull the numbers from the source of record rather than the model's reading, execute the arithmetic with a calculator or a rules engine rather than the model's head and check that the result reconciles before it is accepted. Where a multi-step calculation is required, break it into checkable steps and verify each, because FinQA shows the long chain is where the model fails. The agent can propose the approach and find the relevant data, but the actual number must come from a deterministic computation you can trust, not from a model that gets multi-step arithmetic right less than a quarter of the time.
| Source of the number | Reliability on multi-step finance |
|---|---|
| The model's reasoning | ~22.78% on 3+-step programs |
| Deterministic computation, verified | Exact, reconciled before acceptance |
On FinQA accuracy drops to 22.78% past three steps; a frontier model lifts single-step and leaves the cliff, where cost lives. (arXiv:2109.00122)
Routing the math to a deterministic check is part of what VibeModel does as the Pattern Intelligence Layer. We model the patterns of a financial calculation that must reconcile and where the deterministic check belongs, so the agent's numbers are exact and verified rather than approximately right and compounding.
Frequently asked questions
Does a better model fix the multi-step cliff?
It raises the floor but the multi-step drop persists and exact match is unforgiving. Deterministic computation is what makes the number trustworthy.
Why is exact match the right standard?
Because in finance a number off by a cent or a misclassified transaction is wrong. Approximately right is a failure here.
Can the agent do any of the math?
It can propose the approach and locate data, but the actual arithmetic should run deterministically and be reconciled before acceptance.