Key facts.
- AssistantBench shows agents complete only a small fraction of realistic, multi-factor web tasks, where rare combinations of conditions live. source
- FinanceBench shows even strong models struggle on real, specific domain questions, the kind of rare case that becomes an incident in production. source
Why do rare combinations cause incidents?
Testing tends to be one-dimensional: try each input type, each error, each condition on its own, confirm the agent handles it and move on. Production is multi-dimensional. The conditions arrive together, an unusual input during a degraded dependency while the user does something unexpected and the agent that handled each factor alone meets the combination it never saw and fails. This is why post-incident reviews so often conclude "we tested all of these individually," because the missing test was the interaction, not any single piece. The AssistantBench and FinanceBench results show agents are already shaky on realistic, many-factor tasks, so the combinatorial space where these interactions hide is exactly where reliability is weakest.
You cannot test every combination, the space is too large, but you can hunt the dangerous ones deliberately. Identify the factors that interact, the high-consequence actions, the conditions that change behavior and test their combinations, especially the ones that would be bad if they coincided. Use adversarial and scenario testing to construct the rare co-occurrences rather than waiting for production to assemble them. And design the agent to detect when it is in an unusual combined state and fall back safely, since you will never enumerate them all. The incident you prevent is the combination you tested or the one your fallback caught, not the one you hoped would not happen.
How do you cover the combinations?
| Approach | Test factors alone | Test combinations |
|---|---|---|
| Coverage | Each case once | Dangerous co-occurrences |
| Method | One-dimensional | Scenario and adversarial |
| Unforeseen combo | Found in production | Detected, safe fallback |
AssistantBench finds few many-factor tasks done; each factor handled alone, the untested combination is the incident a better model widens. (arXiv:2407.15711)
Knowing which combinations are dangerous enough to test requires understanding which factors and patterns carry consequence, which the Pattern Intelligence Layer makes explicit. VibeModel surfaces the high-consequence patterns and the conditions that change the agent's behavior, so testing concentrates on the combinations that would actually hurt and the agent can recognize an unusual combined state and fall back, turning rare-combination incidents into cases the system anticipated.
Frequently asked questions
Why test combinations if each case passes?
Because production delivers conditions together and an interaction no one tested is where most incidents originate, even when every factor is handled alone.
Can you test every combination?
No, the space is too large. Hunt the dangerous co-occurrences deliberately and design a safe fallback for the unforeseen ones.
How do you find dangerous combinations?
Identify interacting factors and high-consequence actions, then use scenario and adversarial testing to construct the rare co-occurrences.