Key facts.
- GSM-Symbolic shows reasoning accuracy drops under small perturbations of the same problem, so success under variability is the metric that matters, not best-case accuracy. source
- Reflexion is a feedback-based method whose improvement must be measured to confirm, illustrating why reliability metrics need to capture recovery and improvement over time. source
- GSM-Symbolic drops accuracy under minor perturbations, so a good-day number lies; a stronger model shifts the average, not the fragile cost case. (arXiv:2410.05229)
Why is best-case accuracy the wrong metric?
A single headline accuracy answers "how well can the agent do on a good run," which is the least useful question for production. What you actually need to know is how the agent behaves when conditions vary, because production varies constantly. The GSM-Symbolic result is the warning: take the same problem, perturb it slightly and accuracy drops, which means a best-case number can hide a system that is brittle to exactly the variation production supplies. Optimizing toward the headline number can even make this worse, tuning the agent for the clean case while its behavior under variability, the thing that determines real reliability, goes unmeasured and unimproved.
The metrics that matter are agent-specific. Success rate under variability: how often the agent succeeds across perturbed and realistic inputs, not just clean ones. Consistency: how often it produces the same correct result across repeated runs of the same input, since a one-time success is not reliability. Recovery time: how quickly it gets back to correct behavior after a failure or a degraded condition. And the effect of interventions: when you add a feedback method like Reflexion or a verification layer, did the reliability metrics actually move. Measuring these gives you a target that corresponds to production reliability, so improvement effort goes into the behavior that determines whether the agent can be trusted, rather than into a best-case number that looks good and does not survive contact with a varying world.
What should you measure?
| Metric | Best-case accuracy | Reliability metrics |
|---|---|---|
| Success | On a clean run | Under variability |
| Consistency | Unmeasured | Same result across runs |
| Recovery | Ignored | Time to recover after failure |
| Interventions | Assumed to help | Measured effect |
Measuring success under variability and consistency requires a definition of correct behavior to score against across many runs, which is what the Pattern Intelligence Layer provides. VibeModel makes the agent's expected handling of each situation explicit, so reliability metrics can be computed against those patterns under varied conditions, giving improvement work a target that reflects production reliability rather than a best-case figure that flatters the agent.
Frequently asked questions
Why not report accuracy?
Because a best-case accuracy hides brittleness. GSM-Symbolic shows small perturbations drop accuracy, so success under variability is the metric that predicts production reliability.
What is consistency and why measure it?
How often the agent gives the same correct result across repeated runs. A one-time success is not reliability; consistency is.
Why measure intervention effects?
Because methods like Reflexion or verification layers are assumed to help. Measuring reliability metrics before and after confirms whether they actually did.