Key facts.
- Thinking Machines Lab found that even at temperature 0, sampling 1,000 completions yielded 80 distinct outputs, because batch-invariance in the inference kernels isn't guaranteed, formalizing why one run isn't a measurement.source
- MAST shows the variety behind that inconsistency: fourteen distinct failure modes, so "it failed" isn't one number to track but several.source
- OSWorld illustrates the measurement gap on real-computer tasks: in the original 2024 study the best model completed about 12% versus roughly 72% for humans, a number you only learn by measuring at scale.source
- OpenTelemetry GenAI conventions give the per-run substrate to aggregate, so measuring the distribution is an instrumentation choice, not a research one.source
Why does a single run measure so little?
The agent samples its next step from a distribution, so two runs of the same task explore different paths. A success tells you it can do the task. Not that it will. A failure tells you it can fail, not how often. Neither is the thing you need to improve against, which is the rate. The tau-bench pass^k metric exists precisely to capture this: it asks not "did it pass" but "how consistently does it pass," and the answer for frontier models is sobering. Measure once and you will ship on the strength of a run that doesn't generalize.
this is why the flawless demo is dangerous. The demo is a single run, often a lucky one, and it creates confidence that the distribution doesn't support. The teams that avoid this measure the rate before they ship, across enough runs to see the variance, and they instrument production so the rate keeps being measured after launch, because the distribution drifts.
What to measure instead of pass/fail
Consistency, not just success. Run each representative task enough times to get a stable rate. Track that rate, not a single outcome. Measure failure modes separately, because MAST shows they're distinct and a single "failure rate" hides which one is rising. And measure in production continuously, because the rate you validated pre-launch is a snapshot of a distribution that moves as inputs, tools, and models change. The goal is a number that responds when the agent gets better or worse, which a single run never does.
| What you measure | Single run | Distribution |
|---|---|---|
| Tells you | It can pass once | How often it passes |
| Survives non-determinism | No | Yes |
| Improvable against | No | Yes |
One passing run is an anecdote: at temperature 0, 1,000 completions gave 80 outputs (Thinking Machines), so a bigger model adds rework, not less. (arXiv:2503.13657)
The Pattern Intelligence Layer measures at the level where the number is real: the pattern's behavior across runs, not any single output. Consistency, per-mode failure rates, and production drift become standing measurements, so improving the agent means moving a number you can trust. Reliability at the pattern level starts with measuring the right thing, which for a non-deterministic agent is never one run.
Frequently asked questions
How many runs make a measurement?
Enough to stabilize the rate for the variance you observe. The point is that one is never enough; a stable rate is the unit, not a single outcome.
Our demo passed every time. Are we fine?
A demo is a handful of runs, often curated. tau-bench shows consistency across eight tries can fall to a quarter. Measure the rate, not the demo.
Why measure in production if we measured pre-launch?
The distribution drifts as inputs and models change. A pre-launch rate is a snapshot, not a guarantee.