On real computer tasks, the best model solved 12.24% while humans solved 72.36%. OSWorld's 369 real-application tasks expose a gap that tidy benchmarks hide entirely (OSWorld, arXiv:2404.07972, 2024). If your agent only ever met mocked tools and clean fixtures, that 60-point gap is waiting for it in production.
Key facts.
- Real environments are far harder than curated ones: on OSWorld, 369 real-computer tasks, humans complete about 72.36% while the best model at publication managed 12.24%, a gap that synthetic benchmarks hide (OSWorld, arXiv:2404.07972, 2024).
- Tool-selection reliability drops under distribution shift, the gap between curated test inputs and real ones, so a mock-passing agent is untested where it will actually operate (Quantifying Distributional Reliability of Agentic Tool-Selection, arXiv:2510.03992, 2025).
- Synthetic tools and mocked feedback make tasks easier and less realistic, so they act as proxies for capability rather than evidence of production readiness (agent-evaluation engineering practice, 2025-2026).
Why doesn't a passing sandbox predict production?
Because the sandbox removes the exact things that break agents. Mocked tools return idealized, deterministic responses, so the agent never sees a 422 on an odd parameter combination, a partial result, a rate limit, or a timeout. Synthetic data is clean and short-tailed, so the long-tail inputs and schema drift that real users and real systems produce never appear. The agent optimizes for the tidy trajectory, and in production one misparsed result or wrong parameter cascades. OSWorld makes the size of this gap concrete: agents that look capable elsewhere clear only a small fraction of real-computer tasks while humans clear most of them (OSWorld). The test was easier than the job.
What does a mocked tool hide?
Variance. A mock returns the shape you told it to return, so it cannot show you the carrier feed that changed its schema last week, the auth token that expires mid-run, the endpoint that is fast at 10 requests and slow at 500, or the downstream record that is in a state your fixtures never modeled. Synthetic data hides the distribution: the rare account type, the malformed field, the unusual locale. The agent that passed every mocked case is untested against all of it. This is why tool-selection reliability drops under the distribution shift between curated test inputs and real ones, so a mock-passing agent is untested where it will actually operate (arXiv:2510.03992).
How do you build a sandbox that predicts production?
Close the parity gap deliberately. Build eval sets from real, anonymized or production-calibrated data that carries the long tail and the edge cases, not just happy-path fixtures. Test against live or near-live tool endpoints where it is safe, so real schemas, latencies, and error rates are part of the run. Mirror production infrastructure so rate limits and auth behave as they will. Judge on outcomes, the final system state, not just whether a tool call returned without an exception. And keep evaluating after launch with shadow or canary traffic, because behavior drifts as systems and data change, so evaluation has to be continuous rather than a terminal gate. The sandbox then earns the confidence its green check implies.
Sandbox assumption versus production reality
| Sandbox assumes | Production delivers | Fix |
|---|---|---|
| Mocked, deterministic tools | Drift, partial failures, rate limits | Live or near-live endpoints in test |
| Clean synthetic data | Long-tail, malformed, rare cases | Real or production-calibrated eval sets |
| Tool call returned = success | Returned but semantically wrong | Outcome-based validators |
| One-time pass before launch | Behavior drifts over time | Continuous shadow/canary evaluation |
| Single-user, low load | Concurrency, latency under load | Infra parity + load in test |
The pattern is that a sandbox built on mocks and synthetic data tests a friendlier problem than production poses, so a green run predicts very little. Feed it real data, real endpoints, outcome checks, and continuous post-launch evaluation, and the test starts measuring the job the agent will actually do. Making evaluation a continuous, production-grounded function rather than a one-time checkpoint is reliability at the pattern level, which is what VibeModel builds as the Pattern Intelligence Layer.
Frequently asked questions
Are mocked tools useless in testing?
No, they are fine for fast unit checks. They just cannot certify production readiness, because they hide drift, partial failures, and real data variance. Pair them with live-endpoint and real-data evaluation before you ship.
What is the single biggest parity gap?
Data. Synthetic fixtures lack the long tail, so build eval sets from real, anonymized traffic that includes the rare and malformed cases your users actually send.
Why keep evaluating after launch?
Because tools, schemas, and data distributions drift, so an agent that passed at launch can degrade silently. Continuous shadow or canary evaluation catches the drift before a KPI does.