The same prompt gives different answers because models pick each word by sampling from probabilities, and because the servers that run them do not compute exactly the same way twice. You cannot fully switch this off. So stop judging a prompt by one good answer: run it on many inputs, several times each, and decide in advance which kinds of variation your task can tolerate.
Where the variation comes from
A model writes one token at a time. At each step it scores every possible next piece of text and picks one. With sampling on, the pick is weighted by those scores, like drawing from a bag in which likely words hold more tickets. A different draw early in the answer leads to a different sentence, and everything after it follows from that sentence.
The setting that controls how adventurous the draw is is called temperature. Low values favor the most likely next word. High values spread the choice out.
Temperature zero will not save you
The standard advice for consistent answers is to set temperature to zero. In 2026 that advice has three problems.
It never made answers fully repeatable. Anthropic’s API reference says plainly that results are not fully deterministic even at a temperature of 0.0. Researchers at Thinking Machines Lab sampled 1,000 completions of one prompt at temperature 0 from an open model and got 80 different completions, which first parted ways at the 103rd token (Defeating Nondeterminism in LLM Inference). The main cause they found is mundane: servers process your request in a batch with other people’s, the batch size changes with load, and the arithmetic shifts slightly with it.
- completions of one prompt at temperature 0
- 1,000
- different completions came back
- 80
- where they first parted ways
- Token 103
Source: Thinking Machines Lab, Defeating Nondeterminism in LLM Inference.
On many current models you cannot set it. The same Anthropic reference says models released after Claude Opus 4.6 do not support setting temperature, and any value other than 1.0 is rejected with an error. Google strongly recommends keeping temperature at its default of 1.0 for all Gemini 3 models and warns that lower values may cause looping or degraded performance (Gemini 3 developer guide). Developers report the same default-only rule on OpenAI’s reasoning models (OpenAI developer forum).
Where it still works, it hides the problem. If a prompt only gives the right answer when the model is forced down its single most likely path, the prompt is fragile. The next model version will have a different most likely path.
When variation matters
Variation is a feature in some jobs and a defect in others. Sort your prompts into three groups.
- One right answer. Classification, extraction, routing, yes or no decisions, data in a fixed format. Here any variation is a bug. If a ticket is tagged “billing” on one run and “shipping” on the next, the prompt is ambiguous about something, and the fix belongs in the prompt (a sharper definition, an example of the borderline case), not in a setting.
- One right substance, many right wordings. Support replies, summaries, explanations of policy. The wording can vary; the facts, the commitments and the decision cannot. A customer who asks twice and gets two different refund answers has been told two policies.
- Many right answers. Headlines, campaign ideas, first drafts. Variation is the point. Ask for five options in one call instead of re-running for luck.
Why one good answer proves little
Most prompts are approved on one run. Someone tries it, the answer looks good, it ships. That run is one input and one draw. It shows the prompt can produce a good answer, which is a much weaker claim than saying it usually does.
The arithmetic is unforgiving. If a prompt fails one time in ten on some input, a single run misses the problem 90% of the time. Five runs still miss it 59% of the time (0.9 to the fifth power). Twenty runs miss it 12% of the time. Rare failures take volume to find, and production supplies the volume whether you tested or not: at 10,000 conversations a day, a one-in-a-thousand failure happens about ten times a day.
- Fails 1 time in 10
- Fails 1 time in 100
| Fails 1 time in 10 | 1: 90% | 2: 81% | 3: 72.9% | 4: 65.6% | 5: 59% | 6: 53.1% | 8: 43% | 10: 34.9% | 12: 28.2% | 15: 20.6% | 20: 12.2% |
|---|---|---|---|---|---|---|---|---|---|---|---|
| Fails 1 time in 100 | 1: 99% | 2: 98% | 3: 97% | 4: 96.1% | 5: 95.1% | 6: 94.1% | 8: 92.3% | 10: 90.4% | 12: 88.6% | 15: 86% | 20: 81.8% |
Assuming each run fails independently, the chance of seeing no failure is the pass rate raised to the number of runs.
Test across inputs and across runs
There are two kinds of variation, so test against both.
- Across inputs. Many different real cases, which catch the prompt that works on your example and fails on the customer who writes in Spanish, in one line or in a fury. Start with twenty real cases, including the awkward ones, and add one every time something fails in real use.
- Across runs. The same case several times. Pick the five cases that matter most or sit closest to a borderline, and run each five times. For a one-right-answer prompt, the verdict should never change. A case that passes three times out of five is a biased coin, and it points at the part of the prompt to fix.
In Prompt Tester you can add the same input as several cases to see whether its verdict flips, or run the whole suite twice. The Prompt Tester compares versions case by case and flags what each change fixed or broke; treat a single flip as possible noise until a second run confirms it. An eval is only as steady as its scoring, and an AI judge varies between runs too, so use exact checks wherever a rule can say what you need.
Design some variation out
Some variation is cheaper to remove than to test for.
- Ask for structure. A category from a fixed list, or JSON with named fields, leaves less room to wander than free text, and a check can verify it on every run.
- Supply the facts. The dangerous variation is usually in facts the model had to produce itself. Pass in the policy, the price and the order status, and say what to do when something is missing.
- Decide in code when you can. If the rule is “refunds within 30 days of delivery,” compute the answer and pass it in. Let the model explain the decision, not make it.
- Pin the model version, and re-test when it changes. A new model version is a new set of probabilities, so your old results no longer apply.
Key takeaways
- Answers vary because of sampling and because of how servers batch requests. You cannot fully switch it off.
- Temperature 0 was never fully deterministic, and many current models either reject other values or are tuned for the default.
- Decide per prompt whether variation is a bug (one right answer), a risk (one right substance) or the point (many right answers).
- One good run proves the prompt can work, not that it usually does. A one-in-ten failure survives five runs 59% of the time.
- Test across many inputs and repeat the important cases. Remove what variation you can with structure, supplied facts and decisions made in code.