How does it know "king" is close to "queen"?
Embeddings · Beginner · 5 min · embeddings, llm · after Tokens
An embedding is the list of numbers a model uses as a word’s position in a space with thousands of dimensions. Words used in similar company end up near each other, and relationships become directions you can add.
1 Guess first
2 Look
Where words live
1 / 3real gpt2 embedding positions, PCA to 2-D (#220)
Every piece becomes a point in a space with thousands of dimensions — squashed here to two.
Words land in clusters nobody placed, and king→queen is the same arrow as man→woman.
Each piece becomes a list of numbers — hundreds or thousands of them. Treat the list as coordinates and every word is a point. Training nudges the points until words that show up in similar company sit close together.
4 You are the model
Tap a star to see its neighbours. Then drag the green arrow so its tail sits on "man" and watch where the head lands.
Tap a star to see its neighbours. Then drag the green arrow onto "man".5 Check
6 Say it back
7 Mission · 5 min
Get a free embedding key and embed three words: "king", "queen", "banana". Compare the distances (cosine similarity) and see which pair sits closest.
Three quick ones
Drawn at random from this lesson's pool; different next time.
Done?
Sources
- https://arxiv.org/abs/1301.3781 (Mikolov et al., 2013)
- https://jalammar.github.io/illustrated-word2vec/
- Positions in the picture are an illustrative layout until E2b (#220)
Published Oct 20, 2026 · updated Oct 20, 2026