<p><b>Key facts.</b></p>
<ul>
<li>Temperature zero does not mean deterministic: sampling 1,000 completions at temperature 0 of one open model produced 80 distinct outputs, the first divergence at token 103, because inference kernels are not batch-invariant (Thinking Machines Lab, Defeating Nondeterminism in LLM Inference, 2025).</li>
<li>The cause is the serving stack, not your prompt: matrix multiplication, normalization, and attention kernels return slightly different numbers depending on the batch size they run in, and batch size shifts with server load you do not control (Thinking Machines Lab, 2025).</li>
<li>The agent-level symptom is measurable: tau-bench's pass^k, the chance an agent solves the same task on all k tries, falls under 25% at k=8 in retail, so the same agent that passes once often fails on a repeat (Yao et al., <a href="https://arxiv.org/abs/2406.12045" target="_blank" rel="noopener">tau-bench, arXiv:2406.12045</a>, 2024).</li>
</ul>
<h2>Same prompt, same settings, different answer. How?</h2><p>I ran the same prompt at temperature zero and got a different answer the second time. That is not supposed to happen. But it does. Your request gets batched with other users' requests on shared infrastructure. That batch size changes the order of low-level arithmetic operations. Those operations are not perfectly stable across different orderings. Tiny numerical differences stack up until one token flips to a different option. From that token on, two identical-looking requests produce different outputs. Thinking Machines Lab proved this with 1,000 runs of the same prompt at temperature zero. 80 distinct completions. First split at token 103. Temperature was not the variable. Batch size was. You have no control over that batch size on shared API infrastructure.</p>
<p>You send the identical request twice and get two different completions. The usual explanation, that the model is sampling randomly, is not the real story. The deeper cause is that GPU inference kernels are not batch-invariant: the low-level operations that run the model, matrix multiplication, normalization, attention, produce slightly different floating-point results depending on how many requests are batched together at that moment. Floating-point addition is not associative, so the order of the reductions, which the batch size changes, changes the last bits of the numbers. Those tiny differences occasionally flip which token is most likely, and from that token on the two runs diverge. Thinking Machines Lab demonstrated it directly: 1,000 temperature-zero completions of the same prompt yielded 80 distinct outputs (Defeating Nondeterminism in LLM Inference, 2025).</p>
<h2>Why does temperature zero not make it deterministic?</h2>
<p>Temperature zero sets the sampling parameter to deterministic mode: always pick the highest-probability token. But that highest-probability calculation depends on the exact floating-point values coming out of the GPU kernels, and those values shift with batch size. So temperature zero gives you the maximum-probability token given this specific batch configuration, not the same token every time. The two concepts are different. Temperature controls randomness in sampling. Batch composition controls the numerical precision of the probabilities being sampled from.</p>
<div class="fig"><img src="/blog/article16-diagram.png" alt="One request feeding into a shifting server batch, where different batch sizes produce slightly different numerical results that flip a token and fan out into divergent outputs"/></div>
<h2>Why does run-to-run variance wreck your agent?</h2>
<p>Agents run many steps in sequence. Nondeterminism at step 5 means step 6 runs on a different foundation than the test run showed. By step 10 the agent is in a completely different state. tau-bench measured pass-at-k: the probability an agent solves the same task correctly on k independent tries. In retail scenarios it falls below 25% at k=8. The agent that passed once has a 75% chance of failing if you run it again. Running one test is not testing the agent.</p>
<pre style="background:#0d1117;color:#e6edf3;font-family:ui-monospace,Menlo,Consolas,monospace;font-size:0.9rem;line-height:1.65;padding:16px 18px;border-radius:12px;overflow-x:auto;margin:1.8rem 0;border:1px solid #20262e;white-space:pre;"><span style="color:#8b949e;"># One run tells you almost nothing. Measure consistency.</span>
runs = [run_agent(task) for _ in range(8)] pass_8 = all(r.success for r in runs) # did it pass every single time?
# And pin what you can control client.generate(prompt, temperature=0, seed=42)
How do you get reproducibility back?
| Symptom | Cause | Fix |
|---|---|---|
| Temp-0 still varies | Batch-variant kernels | Batch-invariant kernels / deterministic inference mode |
| Cannot reproduce a bug | Run-to-run divergence | Pin seed, request deterministic settings, log full inputs |
| Noisy eval scores | Single-run measurement | Measure pass^k, not pass^1 |
| Divergent agent branch | Early token flip compounds | Verify each step before it advances the loop |
Request reproducible inference when providers offer it: fixed seeds, dedicated compute, or batching controls. Measure consistency across multiple runs instead of single-run accuracy. Validate each step's output before the next one uses it. Design around the variance: check that a step produced an acceptable output, rather than assuming it produced the specific output from your test. Where that variance causes real damage in your workflow and where it is safe to absorb is the pattern question VibeModel builds as the Pattern Intelligence Layer.
<h2>Frequently asked questions</h2>
Isn't this just temperature? Set it to zero.
Temperature zero removes sampling randomness but not the structural variance. The numbers feeding the greedy choice still shift with batch size, so identical temperature-zero runs still diverge, as the Thinking Machines test showed with 80 distinct outputs from one prompt.
Is the model broken if it gives different answers?
No. Each answer can be individually fine. The problem is consistency: you cannot reproduce a result, trust an eval delta, or promise a customer the same answer twice, which is a reliability problem, not a correctness one.
Why does this matter more for agents than chatbots?
Because agents chain steps. A single divergent token early in the run sends the agent down a different branch of tool calls and decisions, so small numerical variance becomes a different outcome, which is why pass^k drops so sharply.
How do I make my evals trustworthy?
Report pass^k across repeated runs instead of a single pass^1 score, pin inference settings, and log full inputs and outputs. That separates a real improvement from run-to-run noise.