Key facts.
- TRAIL (2505.08638, 2025): plan-completion rate 81%, goal-achievement rate 47% - a 34-point gap across 200 multi-step agent task trials.
- The gap is largest in tasks with external state dependencies: file system changes, API side effects, or database writes that appear to succeed but don't persist or propagate correctly.
- Most agentic frameworks expose step-completion hooks but provide no built-in mechanism for goal-state evaluation after all steps complete.
- Plan verification is cheap (did step N execute?); outcome verification is expensive (did the world change as intended?). Teams choose the cheap check and call it done.
The structural difference
Plan verification asks: "Did the agent do what it said it would do?" It checks logs, confirms tool calls fired, and validates that each step returned a success code. it's necessary but not sufficient.
Outcome verification asks: "Is the world in the state we wanted?" It checks downstream systems, reads back persisted data, validates that the user-facing effect is present. it's sufficient - but teams skip it because it requires knowing what "correct state" looks like before the agent runs.
The difference surfaces most painfully in tasks with delayed effects. An agent sends a payment instruction. Payment API returns 200. Plan marked complete. Six hours later, the payment processor rejects it, a validation error not surfaced in the sync response. Plan verification saw success. Outcome verification - checking the settlement record 24 hours later - would have caught the rejection. Neither framework ran the second check.
Where the gap appears
Database writes that appear to succeed but violate constraints at commit time. The INSERT returns OK from the ORM layer; the database rolls back a parent transaction. The agent saw success. The row was never written.
API calls that return 202 Accepted but process asynchronously. The agent moves on, assuming completion. The async job fails silently. No callback was registered. The downstream state never updates.
Multi-step workflows where step N+1 needs the persisted output of step N. Step N looks like it completed. Step N+1 runs on stale or default data, the output wasn't persisted in time. Both steps report success. The composed outcome is wrong.
Designing outcome verification
Outcome verification requires a goal specification written before the agent runs. Not a list of steps - a description of the intended world state after completion. This specification becomes the test that runs after the last step.
For synchronous effects, a read-back check immediately after the write is usually sufficient: insert a record, then query it; send an instruction, then confirm receipt. For asynchronous effects, a polling loop or event listener with a defined timeout is required. For multi-system workflows, each system boundary needs its own state check - not just the final system in the chain.
# Outcome verification pattern def verify_outcome(goal_spec, timeout_s=60): deadline = time.time() + timeout_s while time.time() < deadline: current_state = read_world_state(goal_spec.scope) if goal_spec.satisfied_by(current_state): return VerificationResult.SUCCESS time.sleep(5) return VerificationResult.TIMEOUT # escalate, never silently pass
Verification approach comparison
| Approach | What it checks | Catches | Misses |
|---|---|---|---|
| Step-completion logs | Tool calls fired | Execution errors | Async failures, state mismatch |
| API response codes | Immediate response | Sync rejections | Deferred failures, constraint violations |
| Outcome read-back | Persisted world state | Write failures, async errors | Delayed effects past read window |
| Goal-spec evaluation | Full intended state | All of the above | Nothing detectable at verification time |
The Pattern Intelligence Layer encodes goal specifications alongside action patterns. Each pattern specifies not just what the agent should do, but what the world should look like when it's done. Outcome verification is built into every pattern execution, not something a developer has to remember to add.
Frequently asked questions
Will a more capable model close the gap between steps run and goal met?
TRAIL ran every step 81% of trials, hit the goal 47%; a stronger model leaves that rework. (arXiv:2505.08638)