Why two teams measure wildly different injection rates for the same attack

One report says 41%, another says 84%, a third says 4%. They are all right. The number depends on the model, the defense, and the context.

Short answer

Injection success rates vary enormously because they are not one number, they are a function of three things: the model, the defenses in place, and the attack context. The same study of agentic coding editors measured 41 to 84% depending on the model and category, with some categories above 90% (AIShellJack, arXiv 2509.22040). Add a strong defense and the picture shifts again: Constitutional Classifiers cut jailbreak success from 86% to 4.4% (arXiv 2501.18837). The one constant across every configuration is that the rate never reaches zero. So the useful question is not "what is the injection rate," it is "what is my rate, in my setup, and how do I contain the residual." Injection rate is model times defenses times context: AIShellJack ran 41 to 84%, classifiers cut 86% to 4.4%, no upgrade hit zero, incident holds. (arXiv:2509.22040)

A dial whose injection-rate reading swings widely as model, defense, and context change

You have to measure your own configuration: your model, your defenses, your context.

— from “Why two teams measure wildly different injection rates for the same attack”

Key facts.

Across agentic coding editors and frontier models, injection succeeded 41 to 84%, with some categories above 90% (AIShellJack, 2025).
A strong defense shifts the number sharply: Constitutional Classifiers reduced jailbreak success from 86% to 4.4%, still non-zero (Anthropic, 2025).
Context matters: the attacker's goal, the agent's tools, and the data it can reach all move the rate, so a single headline number is misleading.

Why does the variance matter for your defense?

Because a borrowed benchmark number tells you almost nothing about your exposure. A 4% figure from a heavily-defended lab setup does not apply to your undefended agent with broad tools, and an 84% headline does not credit the defenses you have added. You have to measure your own configuration: your model, your defenses, your context. Then you design for that rate and, because it is never zero, you contain the residual. The variance is the reason generic reassurance is worthless and your own measurement is essential.

Crossing-lines chart showing injection rate shifting as model, defense, and context vary independently

Headline number vs. your number

Borrowed headline	Your measured rate
From someone else's setup	From your model, defenses, context
Misleads either way	Reflects your actual exposure
No plan for the residual	Residual measured and contained

VibeModel's Pattern Intelligence Layer measures injection behavior in your actual deployment and contains the residual it finds, so you are working from your number, not a headline. You set the model, the defenses, and the context; we tell you where you really stand and stop what gets through. The rate varies, the residual is real, and the only number that matters is yours.

Frequently asked questions

So which published number should I trust?
None of them as your rate. Use them to understand the range, then measure your own configuration. The variance is exactly why borrowing a number fails.

Does a low measured rate mean I am safe?
Lower is better, but it is never zero. Contain the residual regardless, because the one attack that succeeds is the one that matters.

Why two teams measure wildly different injection rates for the same attack

Why does the variance matter for your defense?

Headline number vs. your number

Frequently asked questions

Join the discussion