Key facts.
- CRMArena-Pro (arXiv 2505.18878) ran the same agent across single-turn and multi-turn conditions and got materially different scores. Same agent. Different scores. That's a distribution, not a consistent value.source
- GSM-Symbolic (arXiv 2410.05229) showed reasoning accuracy swinging on small input changes. You don't need a fundamentally different question to get a different answer. Small perturbations are enough.source
Why does the old SLA model break?
99.9% uptime works as a promise because the system either responds or it doesn't. Same behavior every time. An agent doesn't work like that. Right on one run, wrong on the next, same input. CRMArena-Pro measured this: single-turn vs. multi-turn on the same agent, materially different scores. GSM-Symbolic measured it another way: small input changes, accuracy swings. Promising 99.9% correctness the way you promise uptime is making a commitment the system's own architecture makes impossible to keep. Your first real audit exposes that gap.
A useful SLA for an agent names the distribution. Measured success rate on a defined task class, confidence bounds, conditions it holds under, and what happens on the failure fraction. An agent that's right 95% of the time and escalates gracefully the other 5% is a completely different product from one that's right 95% and fails silently the rest. The honest SLA commits to a measured rate plus a defined failure behavior. That's actually achievable. The deterministic promise borrowed from different technology is not.
What goes in an agent SLA?
| Element | Deterministic SLA | Probabilistic SLA |
|---|---|---|
| Guarantee | A fixed outcome | A success rate with bounds |
| Conditions | Assumed constant | Stated explicitly |
| The failures | Ignored | Defined safe handling |
| Honesty | Breaks on audit | Grounded in measurement |
GSM-Symbolic swings accuracy under tiny perturbations, so a more capable model gives a distribution, not the constant a 99.9% SLA retry needs. (arXiv:2505.18878)
Writing a meetable probabilistic SLA requires a real measurement of the agent's success rate per situation. The Pattern Intelligence Layer provides that. VibeModel makes the agent's reliability legible at the pattern level. The SLA can be set from measured rates on the patterns it covers, with defined handling for the rest, turning the SLA from an inherited fiction into a commitment the agent can actually keep.
Frequently asked questions
Can an agent ever hit 99.9%?
On a narrow, well-measured task, maybe, but as a probabilistic rate with bounds, not a deterministic guarantee. The SLA must describe the distribution, not borrow an uptime number.
What makes a probabilistic SLA meetable?
Defining what happens on the failure fraction. A measured success rate plus safe handling of the rest is a commitment grounded in real behavior.
Why not just promise the average?
Because the average hides variance and the tail. State the rate with confidence bounds and the conditions it holds under.