Key facts.
- On TravelPlanner, GPT-4 solves roughly 0.6% of realistic planning tasks, frequently because the plan rests on assumptions or constraints that do not hold.source
- The Reversal Curse shows models confidently assert relationships they cannot actually generalize, the kind of invented premise a hallucinated plan is built on.source
Why do hallucinated plans slip through?
GPT-4 solves 0.6% of TravelPlanner tasks because the plan invents its premises; a more capable model invents them more fluently. (arXiv:2402.01622)
An agent's plan reads as coherent because the model is good at producing coherent text and coherence is exactly what fools the reviewer. The plan lays out sensible-looking steps and nobody checks whether the assumptions underneath them are true: that the API supports the operation the plan assumes, that the data has the field the plan references, that the constraint the plan relies on actually holds. The TravelPlanner result shows how often this fails In practice: with realistic plans collapsing because they assumed things that were not so and the Reversal Curse shows the model will confidently assert relationships it cannot back up, which is precisely how a false premise enters a plan. Acting on such a plan means executing confidently toward a goal built on a fact that was never true.
The defense is to verify the assumptions, not the prose. Before executing, surface the premises the plan depends on and check the ones that can be checked: confirm the API supports the operation, that the data exists, that the constraint holds. Where an assumption cannot be verified, treat it as a risk to flag rather than a fact to act on. This is cheap relative to executing a hallucinated plan and discovering the false premise only when an action fails or, worse, succeeds in doing the wrong thing. The plan's fluency is not evidence of its correctness; the truth of its assumptions is and that is what deserves the check.
What do you verify in a plan?
| Focus | Trust the coherence | Check the assumptions |
|---|---|---|
| What is reviewed | The plan reads well | The premises it rests on |
| Checkable facts | Assumed | Confirmed before acting |
| Uncheckable ones | Acted on | Flagged as risk |
| Outcome | Acts on a false premise | Caught before cost |
Checking a plan's assumptions against reality requires knowing which premises matter and what the ground truth is, which the Pattern Intelligence Layer helps provide. VibeModel makes the conditions a correct plan depends on explicit, so the agent's hallucinated assumptions can be tested against the patterns that define a valid plan and a confident plan resting on an invented premise is caught before it turns into a wrong action.
Frequently asked questions
Why does coherence mislead reviewers?
Because models produce coherent text easily, so a plan reads well regardless of whether its assumptions are true. Coherence is not correctness.
What is a hallucinated assumption?
A premise the plan treats as fact that the model invented, like an API capability or data field that does not exist, the kind of confident assertion the Reversal Curse documents.
What if an assumption cannot be checked?
Flag it as a risk rather than acting on it as a fact, so the uncertainty is surfaced before the plan executes.