How does it know what "it" refers to?
Self-attention · Beginner · 5 min · attention, transformer, llm
Self-attention is how a language model decides which other words matter for each word: every word scores every other word, and the scores are learned from text, not written as rules.
1 Guess first
2 Look
Who does "it" point at?
1 / 40%The83%animal0%didn0%'0%t0%cross0%the5%street0%because0%it0%was0%too5%tiredreal bert-base-uncased attention, one head (#220)
The sentence as pieces in a theatre. "it" is down front with the light. No scores yet.
One word changes and the light moves from "animal" to "street": the scores are about the whole sentence, not the nearest noun.
It is not reading left to right like you. All pieces look at all pieces at once, and each pair gets a score. High score = pay attention. Nobody typed the scores in; they came out of reading a very large amount of text.
4 You are the model
You hold the light. Tap the seat where you think it will settle, press Sweep, then flip "tired" to "wide" and try again.
0%ittap the seat where you think the light will settle
5 Check
Why "self"? Because the sentence scores itself: no outside helper, no lookup. That is also why it can follow a paragraph, not just a line — every piece can reach every other piece in one step.
7 Say it back
8 Mission · 2 min
Open a free chatbot. Paste "The animal didn’t cross the street because it was too tired" and ask what "it" refers to. Then paste the "too wide" version. Then try "The dog chased the cat because it was fast" and watch it hedge.
Three quick ones
Drawn at random from this lesson's pool; different next time.
Done?
Go deeper
Sources
- https://research.google/blog/transformer-a-novel-neural-network-architecture-for-language-understanding/ (the animal/street example)
- https://arxiv.org/abs/1706.03762 (Vaswani et al., 2017)
- Weights in the picture are illustrative until E2b (#220) lands GPT-2 small numbers
Published Oct 20, 2026 · updated Oct 20, 2026