How did it learn all this?
Training: loss and gradient descent · Builder · 8 min · training, llm · after Next-token prediction
Training is a loop: the model guesses the next piece of real text, a loss score measures how wrong the guess was, and every number inside is nudged slightly downhill. Billions of small nudges, no rules, no teacher explaining anything.
1 Guess first
2 Look
Guess, measure, nudge
1 / 3learning rate 0.4 · steps 0
The model starts as a ball dropped somewhere random. Height is loss: how wrong its guesses are on the training text.
The ball only ever moves a little downhill, yet it finds the valley — unless the nudge is so big it flies out.
Take a sentence from the training text, hide the next piece, let the model guess. The loss says how far the odds were from the truth. Then nudge every one of the billions of numbers a hair in the direction that lowers the loss. Repeat on the next sentence. For months.
4 You are the model
Choose a nudge size (the learning rate), then Step the ball down the hill. Reach the valley without flying out.
tinyhugePick a nudge size, then step the ball down. Reach the valley without flying out.5 Check
This is why facts can be fuzzy: nothing was stored as a fact. The numbers that make "Paris" likely after "capital of France" are the same numbers that do everything else, nudged together.
7 Say it back
8 Mission · 5 min
Grab a free GPU quota and run a tiny training notebook (any "train a character-level model" tutorial). Watch the loss number fall for five minutes. That falling number is this lesson.
Three quick ones
Drawn at random from this lesson's pool; different next time.
Done?
Go deeper
Sources
Published Oct 20, 2026 · updated Oct 20, 2026