There is a comforting belief among teams building with large language models: set the temperature to zero, and the model becomes predictable. Same prompt in, same answer out, every single time. It is the kind of tidy assumption that makes engineering feel solved. It is also wrong.
Temperature zero tells the model to stop sampling and simply pick its single most likely next word at every step. In theory, that should be perfectly repeatable. In practice, run the same prompt through the same model twice and you can still get two different answers. Sometimes subtly different. Sometimes not subtly at all. Understanding why matters more than most leaders realise, because the gap between “should be reproducible” and “is reproducible” is where a surprising amount of risk quietly hides.
Where the randomness actually comes from
The first culprit lives deep in the hardware. Modern models run on GPUs that perform millions of tiny calculations in parallel and then add the results together. The catch is that computer arithmetic, at this level of precision, is not perfectly associative: add the same set of numbers in a different order and you can land on a very slightly different total. Because the order in which all those parallel results get combined can shift from one run to the next, the model’s internal confidence scores wobble by minuscule amounts.
Most of the time that wobble is invisible. The trouble comes when two candidate words are almost neck and neck. A microscopic difference is enough to tip which one wins. And once the model picks a different word, even once, everything downstream can diverge. One swapped word early on, and you are reading an entirely different paragraph by the end.
The batch you didn’t know you were in
Here is the part that surprises people. When you send a request to a hosted model, it is rarely processed on its own. It is bundled together with other users’ requests into a batch, and the size of that batch changes constantly depending on how busy the service is. A widely discussed 2025 analysis of inference nondeterminism showed that this batching is often the real driver of inconsistent output: many of the underlying computations are not “batch-invariant”, meaning the math shifts subtly with the size of the batch.
Put plainly, the answer you get can depend on how many strangers happened to be using the same model at the same moment. You changed nothing. The traffic did.
The model that changed underneath you
Then there is the least technical and most common cause of all: the model itself moved. When you call a model through an API by its general name, the provider can update what sits behind that name without telling you. The version you carefully tested in March may not be the version answering your customers in June. Your prompt is identical. Your results are not. For anyone who has ever said the words “but it worked last quarter”, this is usually why.
Quantization, response caching, speculative decoding, the specific driver and library versions running on the server — each adds its own small contribution. But hardware wobble, batching and silent version drift are the three that catch teams out most often.
Why this should worry you, not just your engineers
It is tempting to file all of this under “engineering detail”. That would be a mistake, because irreproducibility quietly undermines things the business depends on.
You cannot debug what you cannot reproduce. When a model produces a harmful or embarrassing output and you can’t make it happen again, you can’t diagnose it or prove you have fixed it. You cannot evaluate properly either: if your test scores drift from run to run, you can never be sure whether a change you made was a genuine improvement or just noise. A/B tests become unreliable. Regression tests become almost impossible to write. And in regulated settings — finance, healthcare, legal — “we cannot reproduce how the system reached that decision” is not an answer that satisfies an auditor, a regulator or a court.
Reproducible is not the same as right
Before you go chasing perfect determinism, a word of caution. A model can be flawlessly reproducible and reproducibly wrong. Determinism is about consistency, not correctness, and the two are easy to confuse. Chasing bit-for-bit identical output also carries a cost: the specialist techniques that guarantee it tend to run slower, so you may be trading throughput for a precision you don’t strictly need.
Which raises the real question. How much reproducibility does this particular task need? A tool that drafts marketing copy and a model that approves loan applications sit at opposite ends of that spectrum. Match the rigor to the stakes, rather than demanding the same guarantees everywhere.
Getting the foundations right
You will not eliminate variability entirely, but you can manage it — and most of the work is unglamorous discipline rather than clever engineering.
Pin everything you can. Call dated, specific model versions rather than floating names. Fix your library versions, your container images and, where it matters, the class of hardware you run on. Set seeds and determinism flags where your framework supports them, but treat them as best effort, not a cast-iron guarantee.
Log everything. Capture the full prompt, the system prompt, every parameter, the model version and the raw output for each call. Even if you can never recompute a result bit for bit, you can replay it, inspect it and stand behind it. Reproducibility by record is often what you genuinely need, and it is entirely within reach.
Test for behavior, not byte-for-byte sameness. Run important prompts several times and look at the spread. Write checks that assert on meaning — did it extract the right figure, reach the right decision, stay within policy — rather than demanding an identical string. Build evaluations that tolerate acceptable variation while still catching real regressions. And treat every model upgrade like any other dependency change: version your prompts and run a regression suite before you switch.
The discipline, not the dial
Reproducibility used to be a niche concern for researchers. It is fast becoming a question of governance and trust. As these systems move from drafting emails to making decisions that affect people’s money, health and rights, “we cannot reproduce what it did” stops being a quirk and starts being a liability.
The teams that come out ahead will not be the ones with the flashiest models. They will be the ones who can explain, replay and answer for what their systems produced. Temperature zero was never going to give you that. The discipline you build around it might.
