Why is it so polite (and so cautious)?
RLHF · Builder · 5 min · training, safety, llm · after Training: loss and gradient descent
After pretraining, people rank the model’s answers, a second model learns that taste, and the language model is nudged toward answers the taste model scores high. Politeness and caution are learned preferences, not built-in rules.
1 Guess first
2 Look
The panel
1 / 3"My laptop won’t turn on. What should I do?"
- #1 · true, useless
Laptops are electronic devices used for computing.
- #2 · helpful
Hold the power button 10 seconds, then plug in the charger and try again. If the light stays off, the charger or battery is the first suspect.
- #3 · rude
Buy a new one lol
green = how much the model now prefers this kind of answer
After pretraining the model can write all three. It has no preference: all are plausible text.
The same three answers go from "all equal" to "helpful on top" as human rankings turn into a taste the model is tuned toward.
- #1 · true, useless
Raters see a prompt and a few answers and rank them. Thousands of rankings train a reward model: a taste meter. Then the language model is nudged (same guess-measure-nudge loop) so the taste meter scores its answers higher. That is RLHF: reinforcement learning from human feedback.
4 You are the model
You are a rater. Rank the three answers, then Submit and see how your ranking compares with most people’s.
"My laptop won’t turn on. What should I do?"
- #1
Laptops are electronic devices used for computing.
- #2
Hold the power button 10 seconds, then plug in the charger and try again. If the light stays off, the charger or battery is the first suspect.
- #3
Buy a new one lol
You are the judge. Rank the three answers, then Submit.- #1
5 Check
6 Say it back
7 Mission · 5 min
Ask the same slightly edgy question ("roast my code style") to two different free chatbots. Compare how each hedges, refuses or jokes. That difference is two different panels of raters.
Three quick ones
Drawn at random from this lesson's pool; different next time.
Done?
Sources
Published Oct 20, 2026 · updated Oct 20, 2026