Key facts.
- Thinking Machines Lab found 1,000 runs of the same prompt at temperature zero produced 80 unique completions, caused by batch-variance in reduction kernels, not sampling randomness.source
- Replacing three reduction kernels with batch-invariant versions made all 1,000 runs bitwise identical, showing the variance was a system property, not a model limit.source
- Reliability is a systems property, not a model upgrade: DORA's 2024 report found AI adoption correlated with worse delivery stability and throughput and concluded the fixes were fundamentals like small batch sizes and testing, the system around the model, not the model itself.source
Why is "the model is not smart enough" the wrong reflex?
Because blaming the model stops the investigation. You order an upgrade and move on. The actual failure gets no attention. What really happened? Context degraded. A tool returned garbage. A verification step was missing. Or the system had the Thinking Machines problem: non-determinism in the inference layer itself. The system doesn't reliably compute the same thing twice. A bigger model hides some of this temporarily. Raw capability absorbs a margin of system sloppiness. Then the system problem grows past the margin. The failure returns. Now more expensive, and you're out of bigger models to buy.
The non-determinism finding is the clearest counterargument. Model held constant. Sampling off. Output still varied 80 ways across 1,000 runs. That variance lived in the inference system. Not in model intelligence. No upgrade fixes a system that doesn't compute the same result twice.
Where should you actually look?
At the system. Is context still usable or has it degraded? Are tool results verified or trusted blindly? Does the pipeline vary run to run? Is there an external verification step? Those questions find the fixable failure. A model upgrade is sometimes the right answer. But it should come after the investigation, not instead of it.
| Diagnosis | Typical fix | Holds? |
|---|---|---|
| Model not smart enough | Upgrade the model | Hides the system fault temporarily |
| System fault | Fix context, tools, verification, determinism | Resolves the actual failure |
Finding the real cause is what VibeModel does as the Pattern Intelligence Layer. We model the patterns that separate a model-capability failure from a system failure. You spend your effort where the fix actually lives instead of buying capability to mask a system you never debugged.
Frequently asked questions
If planning fails, is a better model the answer?
Thinking Machines ran one prompt 1,000 times at temp zero, got 80 outputs from batch variance; the upgrade misses it, a model retry fails. (source)
So model upgrades never help?
They do, sometimes. The point is to diagnose first, because most planning failures are systemic and an upgrade only masks those.
How does non-determinism cause planning failures?
If the same input can yield different outputs, plans vary run to run and bugs become hard to reproduce, which looks like the model being unreliable.
What is the first system thing to check?
Whether tool results are verified and the context is still usable. Those two account for a large share of misattributed failures.