Why does the same prompt give different answers?
Temperature and sampling · Beginner · 3 min · sampling, llm
A language model does not pick the best next word; it rolls a loaded die over all possible next words. Temperature reshapes that die: low makes the top choice win almost always, high gives unlikely words a real chance.
1 Guess first
2 Look
The loaded wheel, three ways
1 / 3The cat sat on the ___
floor53%bed24%couch9%ground8%edge5%temperature 0.2
real GPT-2 small odds for this prompt (#220)
Cold (0.2): the balls barely move and nearly all are "floor". Draw five times — same answer every time.
The same odds, three heats: cold gives the same word every draw, hot gives a different one almost every time.
Out of the model comes not a word but odds — say "floor" 8%, "bed" 7%, "couch" 5%, and so on down a list of fifty thousand pieces. Then one piece is drawn. Temperature stretches or squashes those odds before the draw.
4 You are the model
Set the heat so that three draws in a row come out different. Then make it boring: three the same.
The cat sat on the ___
floor26%bed22%couch18%ground18%edge16%coldhotGoal: three different pieces in a row.5 Check
6 Say it back
7 Mission · 5 min
With a free API key, send the same prompt at temperature 0 and at 1.5, three times each. Count how many different answers you get at each setting.
Three quick ones
Drawn at random from this lesson's pool; different next time.
Done?
Go deeper
Sources
- https://huggingface.co/blog/how-to-generate
- https://ai.google.dev/gemini-api/docs/text-generation (temperature parameter)
- Odds in the picture are illustrative until E2b (#220) lands model numbers
Published Oct 20, 2026 · updated Oct 20, 2026