Key facts.
- PlanBench evaluates plan generation, cost-optimal planning, replanning and generalization and finds LLMs ineffective at these even on simple IPC-style domains, with prompting format changing scores but not the underlying limit. source
- A structured output schema reduces parse failures and makes a plan machine-checkable, which is where most of the practical gain from formatting comes from. source
- Few-shot examples bias the agent toward the shape of plans you showed it, which helps on near-distribution tasks and fades as the task drifts from your examples. source
What do you actually gain from structure?
PlanBench: the newer model with a clean template falls short on Blocksworld; prompting shifts the score, not the wall. (arXiv:2206.10498)
Two things and they are both about the system around the model, not the model itself. First, consistency: a fixed plan format means the agent's output lands in the same shape every time, so a validator can read it, a human can scan it and the next step in your pipeline can consume it without guessing. Second, catchability: when the plan is structured, you can write rules that reject an obviously broken plan before it executes. That is real value. It is also exactly why people overclaim. The plan got cleaner, so it feels smarter and the temptation is to stop checking the thing that just got easier to check.
The ceiling is set by the model's grasp of the task, not by the neatness of the template. Few-shot examples nudge the agent toward the kind of plan you demonstrated, which is a gift on tasks that look like your examples and a trap on tasks that do not. The examples narrow the agent's imagination in a way that helps until the problem shifts, at which point the agent confidently produces a well-formatted plan for the wrong situation.
So how should you use them?
Use structure for what it is good at and keep the safety net it does not replace. Pick a plan format that a validator can parse and a person can read at a glance. Choose few-shot examples that cover the boundaries of the task, not just the happy path, so the bias they introduce points somewhere useful. Then keep the check. A structured plan is easier to verify, so verify it, rather than treating the format as the verification.
| What you change | Honest gain | What it does not fix |
|---|---|---|
| Structured plan schema | Consistency, machine-checkable output | Whether the plan is correct for the task |
| Few-shot examples | Better plans near the examples | Plans that drift from the examples |
That is the posture VibeModel builds into the Pattern Intelligence Layer. We treat the plan format and the examples as patterns that make a plan checkable, then we model where the check has to sit, so the gain from structure compounds into reliability instead of false confidence.
Frequently asked questions
Do reasoning models change this?
They raise the floor further, but PlanBench-style gaps persist. The plan still needs validation, especially when the task moves off the examples.
Are few-shot examples ever a net negative?
Yes, when the task drifts far from them. The bias that helps near the examples misleads far from them.
Is a schema enough to call a plan validated?
No. A schema makes a plan parseable, not correct. Validation is a separate check on the plan's content.