Why does it keep repeating itself?
Repetition loops: a language model writes one likely piece at a time, and once a phrase is on the page, copying it becomes likelier each time it appears (a self-reinforcing loop), most of all when it always takes the top piece (greedy decoding, temperature 0); some randomness (temperature or top-p), a frequency or repetition penalty, a new chat and a clear format break the loop · Beginner · 5 min · sampling, llm · after Next-token prediction, Temperature and sampling
A chatbot repeats itself because every phrase already on the page makes that phrase likelier next. Always taking the top word turns this into a loop. Some randomness, a repetition penalty or a new chat breaks it.
1 Guess first
2 Look
How one sentence turns into a loop
1 / 6GPT-2 small, top piece every time
I enjoy walking with my cute dog, but I’m not sure if I’ll ever be able to walk with my dog. 1 I’m not sure if I’ll ever be able to walk with my dog. 2 I’m not sure if I’ll ever be able to walk with my dog. 3 I’m not sure if I’ll ever… 4
Numbers mark each copy of the sentence. GPT-2 also put a blank line before copy 3.
GPT-2 small, greedy, computed 2026-10-24
GPT-2, always taking its most likely next piece, continues “I enjoy walking with my cute dog”. It writes one new sentence, then the same sentence again, and again.
Each copy of a sentence on the page makes the next copy likelier. A model that always takes its top word follows those odds into a loop it cannot leave.
A model writes one piece at a time, reading everything so far. Text already on the page is its strongest hint, so a phrase it just wrote becomes a likely next phrase. Write it again and the hint gets stronger. Researchers call this the self-reinforcement effect: the more copies, the likelier one more.
4 You are the model
You are GPT-2, continuing “I enjoy walking with my cute dog”. Three times, at the same spot in the same sentence, pick the next word. Then see GPT-2’s real odds.
Round 1 of 3. The sentence is new.
I enjoy walking with my cute dog, but I’m not ___
GPT-2 small, greedy, computed 2026-10-24: its real top four words at each spot
You are GPT-2. Pick the next word.5 Check
It is not one odd example. Holtzman et al. (2019) let GPT-2 Large write 5,000 texts. Always taking the top word, 73.66% ended stuck in a loop. With top-p (drawing from the likeliest words that together make 95%), 0.36%; human text, 0.28%. Xu et al. (2022) found the same pull: the more copies, the likelier the next.
Chat apps already roll the dice, so big chatbots rarely loop word for word. The pull stays: the same opener, the same ideas in a long chat. Start a new chat so old copies leave the page, and ask for a clear format. In an API, keep temperature near the default and add a frequency penalty: repeats lose odds.
8 Say it back
9 Mission · 5 min
In ChatGPT Free, ask for 15 names for a coffee shop. Then say “15 more” three times in the same chat and count the names, or name patterns, that come back. Then open a new chat and ask once: “60 coffee shop names, numbered, each starting with a different word, none repeating an earlier one.” Which list repeats less?
Three quick ones
Drawn at random from this lesson's pool; different next time.
Done?
Get an email when new lessons come out
New lessons arrive in the weekly digest, only in a week that has one. You confirm by email first; unsubscribe in one click.
Go deeper
Sources
- GPT-2 small (124M, Hugging Face “gpt2”), greedy decoding (top piece every time) of 60 pieces after “I enjoy walking with my cute dog”, computed 2026-10-24 with scripts/explain-data/repetition.ts (plain forward pass of scripts/explain-data/distill.ts). Odds of “ sure” on copies 1–4: 0.2008, 0.6583, 0.9297, 0.9970; average of the seven pieces “’m not sure if I’ll ever”: 0.2354, 0.4937, 0.7251, 0.9905. After the first “dog.”: “ I” 0.2322, new line 0.1686, “ It” 0.0417, “ So” 0.0360. The role’s top four at “I’m not”: copy 1 sure 0.2008, a 0.1175, really 0.0533, going 0.0438; copy 2 sure 0.6583, a 0.0441, going 0.0234, really 0.0228; copy 4 sure 0.9970, certain 0.0017, clear 0.0001, even 0.0001. The same prompt and greedy loop appear in Hugging Face’s “How to generate text” (2020, model gpt2)
- Same run with frequency penalty 0.5, applied as in OpenAI’s frequency_penalty (log-odds minus 0.5 × times already written, then renormalised): after the first “dog.”, “ I” 0.1001, new line 0.1976; the text continues “I’m a big fan of the “I love walking with my dog” meme. …” without repeating the sentence
- Holtzman, Buys, Du, Forbes and Choi, “The Curious Case of Neural Text Degeneration” (ICLR 2020; arXiv:1904.09751 v2, 2020-02-14, https://arxiv.org/abs/1904.09751, checked 2026-10-24): GPT-2 Large (762M), 5,000 passages of up to 200 tokens. Table 1, Repetition %: Greedy 73.66, Beam b=16 28.94, Top-k=40 t=0.7 8.86, Sampling t=0.9 0.66, Nucleus p=0.95 0.36, Human 0.28. Figure 9: a phrase (minimum length 2) counts as a repetition “when it repeats at least three times at the end of the generation”. Figure 4: “The probability of a repeated phrase increases with each repetition, creating a positive feedback loop.”
- Xu, Liu, Yan, Cai, Li and Li, “Learning to Break the Loop: Analyzing and Mitigating Repetitions for Neural Text Generation” (NeurIPS 2022; arXiv:2206.02369 v2, 2022-10-09, https://arxiv.org/abs/2206.02369, checked 2026-10-24): “there are few consecutive sentence-level repetitions in human corpora (e.g., 0.02% in Wikitext-103)”; “the more times a sentence is repeated in the context, the higher the probability of continuing to generate that sentence”. Table 1 (greedy, Wikitext-103), Repetition-Sen: MLE 14.50%, Human 0.01%
- OpenAI API reference, frequency_penalty (openai/openai-openapi spec, checked 2026-10-24): “Positive values penalize new tokens based on their existing frequency in the text so far, decreasing the model’s likelihood to repeat the same line verbatim.”
- Google, Gemini API docs, “Gemini 3 Developer Guide” (https://ai.google.dev/gemini-api/docs/gemini-3, last updated 2026-10-09, checked 2026-10-24): “we strongly recommend keeping the temperature parameter at its default value of 1.0”; “Changing the temperature (setting it below 1.0) may lead to unexpected behavior, such as looping or degraded performance”
- The mission’s coffee-shop prompt and the habit wording are illustrative, written for this lesson on 2026-10-24
- aifree.dev catalog, chatgpt-free (verified 2026-10-12, live check 2026-10-24): ChatGPT Free Plan
Published Oct 24, 2026 · updated Oct 24, 2026