Key facts.
- ToolEmu, an LM-emulated sandbox, finds agents frequently take risky or irreversible actions in high-stakes scenarios, especially when conditions deviate from the expected. source
- Greshake's indirect prompt injection work shows an unexpected, crafted input can hijack an agent, so inputs cannot be treated as benign by default. source
Why does the unexpected input cause the worst failures?
The unexpected input is where agents act dangerously; an upgrade does not save you, with ToolEmu's risky actions, so default to safe. (arXiv:2309.15817)
An agent built for the expected case has an implicit assumption baked in: inputs will look like the ones it saw in testing. Production violates that constantly, with malformed data, an API that changed its shape overnight, a user request phrased in a way no example covered. Faced with the unfamiliar, an agent does not pause and ask; it proceeds, often confidently and sometimes destructively. ToolEmu makes this concrete by emulating high-stakes scenarios and finding agents take risky or irreversible actions when things go off the expected path. The failure is not that the agent cannot do the normal case; it is that it does not know it has left the normal case.
Some unexpected inputs are not just surprising, they are hostile. Greshake's indirect injection work shows that a crafted input hidden in a document or web page can take over the agent's behavior, which means handling the unexpected is also a security requirement, not just a reliability nicety. The defense is the same in both cases: build the agent to detect when it is outside its known-good range, by unusual inputs, low confidence or unexpected tool results and have it fall back to a safe default and a human rather than improvise. An agent that recognizes the edge of its competence and stops is far safer than one that confidently acts past it.
What does graceful handling require?
| Element | Assumes clean input | Handles the unexpected |
|---|---|---|
| Off-script input | Proceeds confidently | Detected as out of range |
| Hostile input | Obeyed | Caught and contained |
| Response | Risky or irreversible action | Safe default plus escalation |
Recognizing when an input is off the map requires a clear definition of what on the map looks like, which is what the Pattern Intelligence Layer provides. VibeModel defines the patterns the agent handles reliably, so anything that does not match triggers a safe response instead of a confident improvisation, turning the unexpected input from the cause of a risky action into a case the system was designed to catch.
Frequently asked questions
Why not just train on more cases?
You cannot enumerate every unexpected input. The agent needs to detect when it is outside its known range and fall back safely, not rely on having seen everything.
Is the unexpected input a security issue too?
Often yes. Greshake's work shows crafted inputs can hijack an agent, so handling the unexpected protects both reliability and security.
What is the safe response?
A conservative default plus escalation to a human, rather than a confident action, since ToolEmu shows agents take risky steps when off-script.