Why can’t it count syllables in a poem?
Phonology in text-only LLMs: a model gets numbered pieces of spelling (tokens), never sound, while syllables and rhymes are about pronunciation; English spelling does not match sound (though, through, tough) and tokens split across syllables, so syllable counts and rhymes drift; splitting each line into syllables, a pronunciation dictionary or tool, and checking aloud help · Beginner · 5 min · tokens, llm · after Tokens, Length control in LLMs: tokens are not words, and nothing counts while it writes
A chatbot counts syllables badly because it never hears words: it gets numbered pieces of spelling, and English spelling hides sound. In one test GPT-4 got a sentence’s syllable count exactly right 23.3% of the time.
1 Guess first
2 Look
Spelling in, sound left out
1 / 7model getsyou hear (dictionary)though#5495DH OW like “go”through#1819TH R UW like “true”tough#15500T AH F like “stuff”bough#287 #1870B AW like “cow”cough#62215K AA F like “off”dough#41613D OW like “go”tokens: o200k_base (GPT-4o’s tokenizer), tiktoken 0.14.0; sounds: CMU Pronouncing Dictionary (cmudict 1.1.3); 2026-10-24
Six words that end in “ough”. The model gets a number for each piece of spelling. You hear six different sounds; the numbers say nothing about any of them.
The model gets numbers for pieces of spelling. Syllables and rhymes live in the sound, and English spelling does not show it.
A chatbot reads tokens: numbered pieces of spelling. It has never heard “tough” or “though”, only seen them. Whatever it knows about sound it picked up from text that talks about sound, like dictionaries and poems. So it guesses syllables from spelling, and the guess slips on every word like “fire” or “bough”.
4 You are the model
You are the chatbot. Three rounds: pick the word that rhymes, then the line with exactly 5 syllables. A pronunciation dictionary checks each pick.
Round 1 of 3.
Pick the word that rhymes with “home”.
answers checked by code against the CMU Pronouncing Dictionary (cmudict 1.1.3)
You are the chatbot. Pick an answer; the dictionary checks it.5 Check
The numbers back it up. In PhonologyBench, GPT-4’s syllable counts were exactly right for 23.3% of sentences, people’s for 90.0%. Its rhymes were real 69.1% of the time for common words and 46.1% for rare ones. A 2025 rerun found gpt-4o at 20.8% on counting.
What helps: split, then count. Ask it to write each line with hyphens between syllables, count each line and fix the ones that are off; in the 2025 study, a step-by-step prompt lifted Claude 3.5 Haiku from 21.1% to 57.4%. Give it a pronunciation dictionary or a syllable counter. And clap it out yourself.
8 Say it back
9 Mission · 5 min
In ChatGPT Free, ask for a haiku about autumn rain and clap out each line: is it 5-7-5? Then ask: “Rewrite it. Write every word split into syllables with hyphens, count each line, and fix any line that is not 5, 7 or 5.” Clap it again. Which one was right?
Three quick ones
Drawn at random from this lesson's pool; different next time.
Done?
Get an email when new lessons come out
New lessons arrive in the weekly digest, only in a week that has one. You confirm by email first; unsubscribe in one click.
Go deeper
Sources
- Suvarna, Khandelwal and Peng, “PhonologyBench: Evaluating Phonological Skills of Large Language Models” (arXiv:2404.02456 v2, 2024-04-05, https://arxiv.org/abs/2404.02456, checked 2026-10-24): zero-shot, 1,000 sentences from Romance Books and Reddit Short Stories, gold counts from a grapheme-to-phoneme library; accuracy = sentences whose count is exactly right. Table 4, syllable counting, Overall: LLaMA-2-13B-Chat 6.9, Mistral-7B-Instruct 6.8, Mixtral-8X7B-Instruct 6.6, GPT-3.5-Turbo 19.6, GPT-4 23.3, Claude-3-Sonnet 55.3, Vowel Baseline 12.1, Human 90.0. Table 5, rhyme word generation success rate (five rhymes per word, gold from WordHippo), Common/Rare: GPT-3.5-Turbo 66.5/42.7, GPT-4 69.1/46.1, Claude-3-Sonnet 62.4/39.6, Human 86.4/60.4. Abstract: “Despite having no access to speech data”
- Jang, Ahn and Shin, “P-CoT: A Pedagogically-motivated Participatory Chain-of-Thought Prompting for Phonological Reasoning in LLMs” (arXiv:2507.16656 v1, 2025-07-22, https://arxiv.org/abs/2507.16656, checked 2026-10-24): PhonologyBench syllable counting, 993 sentences, exact match. Table 5, Baseline then best P-CoT: Claude 3.5 Haiku 21.1, 57.4 (P-CoT3); gpt-3.5-turbo 16.0, 48.8 (P-CoT1); gpt-4o 20.8, 22.2 (P-CoT3); Claude 3.5 Sonnet 20.2, 32.6 (P-CoT5)
- Tokens measured 2026-10-24 with tiktoken 0.14.0, encoding o200k_base (GPT-4o), each word with its leading space: “ though” [5495], “ through” [1819], “ tough” [15500], “ bough” [287, 1870] = “ b” + “ough”, “ cough” [62215], “ dough” [41613], “ beautiful” 1 token, “ strawberry” 1, “ every” 1, “ fire” 1, “ syllable” 2 (“ syll” + “able”), “ haiku” 2 (“ ha” + “iku”). Haiku lines: “The quiet ocean” 3 tokens, “Whispers its beautiful secrets” 5, “To the lighthouse dome” 4
- Pronunciations from the CMU Pronouncing Dictionary via the Python package cmudict 1.1.3, read 2026-10-24 (ARPAbet; vowels carry a stress digit, one per syllable): though DH OW1, through TH R UW1, tough T AH1 F, bough B AW1, cough K AA1 F / K AO1 F, dough D OW1; beautiful 3, strawberry 3, every 3 or 2 (EH1 V ER0 IY0 / EH1 V R IY0), fire 2 or 1 (F AY1 ER0 / F AY1 R), syllable 3, haiku 2; the haiku lines 5, 8, 5. Rhyme = the sounds from the last stressed vowel on, computed by code in the lesson
- The haiku draft, the role’s words and lines, the mission prompt and the habit wording are illustrative, written for this lesson on 2026-10-24; their syllables and rhymes are checked by code against the dictionary entries above
- aifree.dev catalog, chatgpt-free (verified 2026-10-12, live check 2026-10-24): ChatGPT Free Plan
Published Oct 24, 2026 · updated Oct 24, 2026