Key facts.
- GAIA shows humans at 92% and GPT-4 with plugins at 15% on conceptually simple real-world tasks, so the agent's first plan is often wrong in ways a dry run would expose. source
- Simulating a plan against a model of the world predicts the outcome cheaply, before any irreversible action fires. source
- A sandbox execution catches state and side-effect problems that a purely mental simulation misses, making the two complementary. source
What does simulating a plan buy you?
A free wrong answer. Execution is where a bad plan costs real money and real side effects and most of those costs are avoidable if the agent checks the plan against a model of the world before committing. The agent asks, in effect, "if I run this, what happens," and reasons through the predicted result. If the prediction is bad, the plan never executes and you paid for a thought instead of a failure. For higher stakes, a sandbox makes the dry run concrete: run the plan against a copy of the environment, observe the actual state changes and only promote the plan to the real system if the sandbox outcome is what you wanted.
The GAIA result is the reason this matters more than it sounds. An agent that scores 15% where humans score 92% is producing first plans that are frequently wrong on tasks that are not even hard. Letting that first plan execute directly is the expensive way to discover it was wrong. Simulating it first is the cheap way.
Mental simulation or sandbox?
Both, by stakes. A lightweight mental simulation, the agent reasoning through the outcome, is cheap and catches obvious logical faults. A sandbox execution is heavier and catches the state and side-effect problems that reasoning alone misses, which is what you want before anything irreversible. Match the rigor of the dry run to the consequence of the action and reserve the real execution for plans that already passed a cheaper version of themselves.
| Dry run type | Catches | Use when |
|---|---|---|
| Mental simulation | Logical and coherence faults | Low to moderate stakes |
| Sandbox execution | State and side-effect faults | Before irreversible actions |
GAIA defeats GPT-4 plugins at 15% to humans' 92%; a stronger model that runs its first guess live skips the dry run and eats rework. (arXiv:2311.12983)
Knowing which plans need which dry run is part of what VibeModel does as the Pattern Intelligence Layer. We model the patterns of a plan worth simulating and the rigor the stakes demand, so the failures surface in a dry run instead of in production.
Frequently asked questions
Is mental simulation reliable?
Imperfectly, but it catches obvious faults cheaply. For anything irreversible, back it with a sandbox.
Doesn't a sandbox double the work?
It does a fraction of the work of cleaning up a bad production action. The trade favors the sandbox on high-stakes plans.
What needs an accurate world model?
The sandbox does. Keep it close enough to production that a passed dry run actually predicts the real outcome.