Key facts.
- GAIA's questions require reasoning, multimodality, browsing and tool use and are simple for humans (92%) But defeat GPT-4 with plugins (15%), inverting the usual story of models beating people on professional exams.source
- WebArena shows the same pattern on realistic web tasks, with the best agent at 14.41% against 78.24% for humans, the gap widening on tasks the agent has no template for.source
- Production teams respond by bounding autonomy: in a 2025 study of deployed agents, 68% execute at most ten steps before a human checkpoint, which keeps novel cases from running unchecked.source
- GAIA: humans 92%, GPT-4 plugins 15%; the upgrade only retries an improvised plan, it does not rescue it. (arXiv:2311.12983)
Why is "novel and easy for a human" the worst case?
Because the agent has no reason to hesitate. A hard, specialized task at least signals its difficulty and the agent often hands it off or refuses. A task that is novel but superficially simple looks like something the agent can do, so it produces a confident plan that is subtly built for the wrong problem. The model is pattern-matching to the nearest thing it has seen and when the real task sits in a gap between those patterns, the match is plausible and wrong. Humans handle these by noticing the request is unusual and adapting. Agents tend not to notice.
This is what out-of-distribution means In practice: . It is not exotic. It is the ordinary, slightly-off request that does not match the shape of anything the agent was built to do, arriving with no flag that says "this one is different."
What works instead of hoping for adaptability?
Define the space. Be explicit about the kinds of tasks the agent is meant to handle and build a detector for requests that fall outside it. When a task is out of distribution, the right move is not a brave improvised plan, it is an escalation or a request for help. The teams whose agents survive contact with real users are the ones that made the agent good at one characterized space and honest about its edges, rather than mediocre and overconfident across an open one.
| Posture | Novel-task behavior | Outcome |
|---|---|---|
| Open-ended autonomy | Improvises a confident plan | Plausible, wrong, unflagged |
| Bounded scope plus detection | Flags the task as out of range | Escalates instead of failing silently |
VibeModel works at exactly this seam. As the Pattern Intelligence Layer, we characterize the patterns an agent reliably handles and the patterns that fall outside them, so the agent knows the edge of its competence rather than discovering it in production.
Frequently asked questions
Won't a more capable model generalize better?
Somewhat, but GAIA shows the gap is large on tasks humans find trivial. Detection of out-of-range requests matters more than raw capability.
How do I detect out-of-distribution tasks?
Characterize the in-scope space and score new requests against it. Low similarity is your signal to escalate.
Is narrowing scope a permanent limit?
No. You widen the characterized space deliberately as you learn it, rather than leaving the agent open to everything at once.