Key facts.
- Models hallucinate partly because evaluations reward guessing over expressing uncertainty, so a model asked to rate its own plan tends to overstate confidence. source
- Plan quality, scored on coherence, constraint satisfaction and coverage, correlates with outcome and arrives before execution, making it a usable leading indicator. source
- Realigning evaluation to reward uncertainty, rather than penalize it, is the proposed fix and applies directly to how you score plans. source
Why measure the plan instead of the outcome?
OpenAI shows models trained to guess, so a self-graded plan score is a more capable model saying sure, no data. (arXiv:2509.04664)
Because the outcome is a lagging indicator. By the time the agent's result is wrong, it has already spent the time, the tokens and the side effects. A plan-quality score lands before any of that, while you can still stop a bad run. If you can tell that a plan is incoherent, violates a constraint or skips a required step, you have a signal you can act on and you got it for the price of evaluating text rather than executing it. That is the whole appeal of treating plan quality as a leading indicator: it moves the detection point upstream of the cost.
The trap is letting the planner score itself. A model trained to be a good test-taker will rate its own plan highly, because guessing confidently is what the training rewarded and admitting doubt is what it penalized. So the self-score is reliably optimistic, exactly when you needed it to be honest. The measurement has to come from somewhere the planner does not control.
What does an honest plan-quality metric look like?
It is scored by something other than the planner, on axes that map to real failure: does the plan cohere, does it satisfy the stated constraints, does it cover the required steps and does it flag where it is unsure. An uncertainty signal counts in its favor, not against it, which is the inversion the hallucination work argues for. A plan that says "I am not confident about this branch" is more useful than one that is smoothly, wrongly certain.
| Who scores the plan | What you get | |
|---|---|---|
| The planner itself | An optimistic guess dressed as a score | |
| An independent evaluator | A leading indicator you can act on |
Building that independent measurement is part of what VibeModel does as the Pattern Intelligence Layer. We model the patterns of a sound plan and score against them from outside the planner, so plan quality becomes an early warning you can trust rather than a confidence the model wanted you to share.
Frequently asked questions
Can the same model score a different plan?
Better than scoring its own, but self-preference effects persist. An independent evaluator or rules-based check is stronger.
What axes matter most?
Constraint satisfaction and coverage, because they map directly to the failures that cost you in production.
Does rewarding uncertainty make the agent timid?
It makes the agent honest about its edges, which is what lets you route the uncertain cases to a check instead of into production.