Key facts.
- OSWorld (arXiv:2404.07972) benchmarks agents on 369 computer tasks across multiple OS environments and measures end-state correctness rather than step completion, finding frontier model success rates of 12-38% on tasks where human success rate is 72-89% (OSWorld, arXiv:2404.07972).
- OSWorld's task evaluation design explicitly captures plan-completion/goal-completion divergence: an agent that executes all planned steps but produces an incorrect end state is scored as failed, even if every step completed without error.
- The most common divergence pattern in OSWorld is "local success, global failure": each individual step succeeds according to its own completion condition, but the combination of steps does not produce the required end state because the agent's mental model of what "done" means differs from the task specification.
- GAIA (arXiv:2311.12983) similarly found that on complex multi-step tasks, agents that attempt all required steps still fail to complete correctly at significantly higher rates than tasks where agents complete fewer steps but verify end state after each one.
- End-state verification, where the agent explicitly checks the final state against the original task specification before declaring completion, improves task success rates in evaluation environments but is rarely implemented in production agent systems.
What creates the plan/goal gap
OSWorld puts plan completion 30 points above correct end state; even frontier models finish every step, miss the goal, rework lands late. (arXiv:2404.07972)
When an agent constructs a plan to achieve a goal, it translates the goal into a sequence of steps. Each step has a completion condition. When the step completes, the agent moves on. At no point does the standard planning architecture check whether the cumulative effect of all steps so far matches what the goal actually required. The agent is tracking plan execution, not goal proximity.
The divergence accumulates through translation errors: the goal is "save the file as a PDF in the correct folder," the plan is "export file as PDF, move to destination folder." The export step completes. The move step completes. But the export was to the wrong format (PDFA instead of PDF), and the folder path was slightly wrong due to a case difference in the OS. Both steps "completed." The goal was not achieved. An end-state verification step - does the destination folder contain a file matching the expected specification? - would catch both failures before the agent declares success.
Implementing end-state verification
End-state verification requires the agent to query the state of the system after plan execution and compare it against a success specification that was defined when the goal was stated - not when the plan was built. The distinction matters: the plan's completion conditions are proxies for success. The original goal statement is the actual success criterion. For computer tasks, end-state verification is often a file check, a UI element check, or an API query. For business process tasks, it is a record query or a business rule validation. In both cases, the check runs after plan execution and before the agent declares success to the user.
Plan completion vs goal completion by task type
| Task type | Typical plan completion rate | Typical goal completion rate | Primary divergence source |
|---|---|---|---|
| File management (OSWorld) | 75-85% | 12-38% | Format, path, and naming mismatches |
| Web form submission | 80-90% | 50-65% | Field validation failures after submit |
| Multi-system data sync | 85-95% | 40-60% | Sync latency and partial record transfer |
| Document editing | 70-80% | 25-45% | Format requirements missed during edit |
VibeModel's Pattern Intelligence Layer learns which plan patterns in your deployment consistently show plan/goal divergence - the cases where all steps complete but the task outcome fails validation. By correlating plan execution traces with post-session outcome checks, it identifies the specific step sequences where end-state verification would prevent the most failures. That turns "add verification" from a generic recommendation into a specific engineering task: add an end-state check for these three plan patterns that have the highest divergence rate in your environment.
Frequently asked questions
What does a good success specification look like for an agent task?
A success specification states the required end state in terms that can be checked programmatically: "The file 'report.pdf' exists in folder '/Reports/2026/Q1/' and is a valid PDF document with page count greater than 0." Not "the report was saved" - that is a plan completion condition. The end state is what should be true in the world when the goal is achieved.
Does OSWorld's performance gap reflect current production agent performance or benchmark performance?
OSWorld evaluates agents on carefully curated tasks with clear success criteria and a controlled environment. Production environments are typically messier, with more ambiguous task specifications and less consistent starting states. The gap in production is at least as large as OSWorld shows, often larger, because production task specifications are more varied and success criteria less explicit.
Can end-state verification be automated, or does it require human review?
For well-specified task types, it can be fully automated: query the expected state, compare to the success specification, flag mismatches. For novel or complex tasks, a hybrid approach works: the agent generates a self-assessment checklist against the original goal, a human spot-checks the flagged items. The automation handles volume; the human handles the edge cases.