Why does it miscount the r’s in "strawberry"?
Tokens · Beginner · 3 min · tokens, llm
A token is the unit of text a language model actually reads: a word, part of a word, or punctuation, each turned into a number. Models count, price and limit everything in tokens, and never see the letters inside one.
1 Guess first
2 Look
"strawberry", two ways
1 / 3strawberryletters10r's3pieces—Your eyes: ten letters, three of them r. Easy.
Switch eyes: the letters you count vanish, and what is left is numbered pieces with the r’s hidden inside.
Before anything else, text is chopped into pieces by a fixed recipe and each piece becomes a number. Common words stay whole; rare words get cut. The model never sees letters — so "count the r’s" is a question it has to answer blind.
4 You are the model
Type any word. Guess how many pieces the model will see, then switch eyes and compare. Try your name, then a long rare word.
raspberryletters9r's3pieces—Guess how many pieces the model sees, then switch eyes.5 Check
6 Say it back
7 Mission · 2 min
Ask a free chatbot how many r’s are in "strawberry". Then ask it to spell the word with one letter per line first and count again. Watch the second answer land.
Three quick ones
Drawn at random from this lesson's pool; different next time.
Done?
Go deeper
Sources
- https://platform.openai.com/tokenizer
- https://ai.google.dev/gemini-api/docs/tokens
- Pieces come from a demo tokenizer until you type, then from GPT-2’s tokenizer reduced to its first 16,000 of 50,000 merges (rare words may split finer); other models cut differently
Published Oct 20, 2026 · updated Oct 20, 2026