Key facts.
- The generator-verifier gap shows a model's ability to verify differs from its ability to generate, so test coverage must verify outcomes independently, not trust the agent's self-report. source
- FActScore demonstrates fine-grained, atomic factual evaluation, the level of rigor production testing needs beyond a coarse pass/fail. source
Why is a handful of tests not enough?
For deterministic code, one passing test per path is meaningful, because the path behaves the same every time. For an agent, a single passing run is one sample from a distribution and the distribution is what you actually ship. Enterprise rigor means running each case many times to estimate the real success rate and its variance and running many cases to cover the input space, including the messy ones. Anything less leaves you with a green light that reflects luck on a few friendly inputs rather than evidence about how the agent behaves at scale. Coverage and statistical rigor are not optional polish; they are what turns testing into a basis for an SLA.
The other half is verifying the right way. It is tempting to scale testing by letting a model judge the outputs, but the generator-verifier gap warns that verification is its own imperfect skill, so a model grading itself or its sibling carries systematic blind spots. Rigorous coverage verifies against ground truth and outcomes, with the granularity that methods like FActScore bring, where a long answer is checked claim by claim rather than waved through as roughly right. The combination, broad coverage, many runs and outcome-grounded verification, is more work than teams expect and it is the difference between trusting an agent because you measured it and trusting it because it passed a demo.
What does real coverage require?
| Dimension | Happy-path check | Production rigor |
|---|---|---|
| Runs per case | One | Many, for a rate and variance |
| Cases | A few friendly ones | Broad, including messy |
| Verification | Agent's self-report | Outcome and ground truth |
| Result | A lucky green light | A defensible SLA |
Coverage is only meaningful if you know what correct behavior is for each case, which is what the Pattern Intelligence Layer defines. VibeModel makes the expected handling of each situation an explicit pattern, so a test suite can measure the agent's success rate against those patterns across the input space, giving you coverage that means something rather than a pile of self-graded passes.
Frequently asked questions
Can a stronger model just verify its own runs?
Verifying is a separate skill from generating, the generator-verifier gap; a more capable model cannot grade itself, so the late incident hides. (arXiv:2505.08827)
How many runs are enough?
Enough to estimate the success rate and its variance with confidence, which for a non-deterministic agent is many per case, not one.
Can a model verify the tests?
Carefully and not alone. The generator-verifier gap means model verification has blind spots, so ground-truth and outcome checks are needed.
Why is granularity important?
A coarse pass/fail hides partial errors. Fine-grained evaluation like FActScore catches the claim-level mistakes a whole-answer check misses.