Key facts.
- AgentDojo (arXiv:2406.13352) puts tool-using agents in realistic tasks with injected attacks and shows that combining read access with action capability is what an attacker exploits, so risk has to be assessed across the whole capability set, not part by part. source
- Greshake and colleagues demonstrated indirect prompt injection, where untrusted content an agent retrieves becomes instructions it follows, a risk that only appears when read access and action capability are combined. source
- On GAIA, a benchmark of real-world assistant tasks, the best evaluated AI assistant scored about 15% against a human baseline near 92%, so an agent wielding a broad capability set will misuse it often enough that the combined risk is real. source
Why does component-level review miss the real risk?
Let one agent read untrusted content, hold data and send messages and it risks exfiltration though each looked fine; a stronger model does not remove it, assess before you build. (arXiv:2406.13352)
Because the risk is not in any component; it is in the chain. A tool that reads documents is safe. A permission to send email is safe. A connection to a customer database is safe. Put all three in one agent and you have built a path where a document carrying a hidden instruction can make the agent read a customer record and email it out and every individual review signed off because every individual capability was reasonable. Greshake's work on indirect prompt injection is the canonical version of this: the danger appears precisely when an agent that ingests untrusted text also holds the ability to act and neither capability is dangerous without the other.
A more capable model does not dissolve the composition risk, because the model is the thing being steered. Mitigations reduce injection success but do not eliminate it and the agent's baseline reliability is low enough on real tasks that misuse of a broad toolset is not a rare event. GAIA measured the best evaluated AI assistant at about 15% on real-world assistant tasks against a human 92%, which is the rate at which a capable, broadly-permissioned agent will do the wrong thing with the tools you gave it. Assessing risk at the composition level, asking what this specific combination of capabilities makes possible, is the only way to see the exfiltration path, the privilege chain or the irreversible action that no single component review would flag.
What does composition-level risk assessment require?
Enumerate the agent's full capability set, then reason about the chains. Which read sources are untrusted? Which actions are irreversible or exfiltrating? Which combinations connect an untrusted input to a high-consequence output? Each dangerous chain gets a control: a boundary, an approval gate, a scope reduction or a separation so the two capabilities never live in the same agent. This is the same discipline AgentDojo's attack model exposes, applied to the specific reality that an agent is a set of capabilities a single instruction can orchestrate.
| Capability A | Capability B | Emergent risk |
|---|---|---|
| Reads untrusted content | Sends messages | Data exfiltration |
| Holds elevated credentials | Executes dynamic commands | Privilege escalation |
| Writes to production | Acts without confirmation | Irreversible damage |
The Pattern Intelligence Layer is where composition risk becomes visible and controlled. The combinations an agent can chain, untrusted input to powerful action, are tracked at the pattern level, so a dangerous chain is flagged and gated even though each capability passed its own review. Reliability at the pattern level is what makes a risk assessment see the whole agent, not a list of safe parts.
Frequently asked questions
Can't I just review each tool and permission carefully?
That misses the chains. The exfiltration and escalation paths live in the combination, which is exactly where component-by-component review is blind. Assess the composition.
Does a safer model remove the need for composition analysis?
No. Injection mitigations lower but do not zero the risk and the agent's task reliability is low, so a broad capability set will be misused. The combination still needs controlling.
What is the cheapest control for a dangerous chain?
Separation. If two capabilities are only dangerous together, keep them in different agents or behind a gate so the chain cannot form in one run.