Why does it drop rules when I give it many?
Instruction following under many constraints: every requirement in a prompt is one more check the reply must pass while it is written token by token, so the chance of keeping all of them falls as they pile up; requirements late in a long list, in conflict, or about counting (exactly N words, never a letter) are dropped most; fewer requests per turn, a numbered list, must-haves first and last and a check against the list help · Beginner · 5 min · context, llm · after System prompts and instruction priority, Lost in the middle: position bias in long contexts
A chatbot drops part of a long request because every requirement is one more check its reply must pass. Each check usually passes, but misses add up: GPT-4 kept all of three instructions about half the time.
1 Guess first
2 Look
Eight requests, one reply
1 / 6You: Write a product blurb for our steel water bottle. Keep it under 60 words, friendly, and in British English. Mention that it keeps drinks cold for 24 hours, that it costs £25 and that it is dishwasher safe. Don’t use any commas. Don’t use exclamation marks. End with a question.
Chatbot: Meet your new favourite bottle. Our steel bottle keeps drinks ice-cold for 24 hours, so your water tastes fresh from morning to night. All this for just £25. Ready to swap your plastic for good?
- ✓ under 60 words
- ✓ friendly
- ✓ British English
- ✓ cold for 24 hours
- ✓ costs £25
- ✗ dishwasher safe
- ✗ no commas
- ✓ ends with a question
illustrative: a request and reply we wrote, not a recorded model answer
One message, eight requirements. The reply reads well and keeps six. “Dishwasher safe”, in the middle, is gone, and there is a comma (illustrative).
Each requirement usually holds on its own. Together they rarely all hold, and the ones in the middle, at the end of a long list or about counting go first.
A model writes its reply one token at a time, and every requirement is one more check each part must pass. It has no checklist beside it to tick off. Each check usually passes, but the misses add up: 0.81 × 0.81 × 0.81 is about 0.53. Eight requests in one paragraph almost guarantee one slips.
4 You are the model
You are the chatbot. Three times, pick the one line of a bike ad that keeps every rule. The rules pile up: one, then three, then five.
Round 1 of 3. Write one line of an ad for a folding bike. Rules:
- 1. mention “bike”
illustrative: the rules and lines are ours, checked by code
You are the chatbot. Pick the line that keeps every rule.5 Check
Which ones drop? In IFScale (2025), with 250 requests at once, most models missed the end of the list more than the start. Rules that conflict, like 5 paragraphs in under 20 words, cannot both hold. And counting is hardest: GPT-4 kept “a letter N times” 60.6% of the time and “no commas” 66.7%, against 97.4% for “include this word”.
What helps: fewer requests per message, splitting a big job into turns. A numbered list, one rule per line. The must-haves first and last, not in the middle. And a closing line: “Before you answer, check your draft against each numbered rule.” For exact counts, count yourself or use a word counter.
8 Say it back
9 Mission · 5 min
In ChatGPT Free, paste the bottle request from the picture as one paragraph and tick off its eight requirements in the reply. Then open a new chat, send the same eight as a numbered list with the must-haves first and last, and end with “Check your draft against each numbered rule before you answer.” Which reply kept more?
Three quick ones
Drawn at random from this lesson's pool; different next time.
Done?
Get an email when new lessons come out
New lessons arrive in the weekly digest, only in a week that has one. You confirm by email first; unsubscribe in one click.
Go deeper
Sources
- Zhou, Lu, Mishra, Brahma, Basu, Luan, Zhou and Hou, “Instruction-Following Evaluation for Large Language Models” (IFEval; arXiv:2311.07911 v1, 2023-11-14, https://arxiv.org/abs/2311.07911, checked 2026-10-24): 25 verifiable instructions, 541 prompts with one to three each. Table 3, GPT-4: prompt-level strict-accuracy 76.89, inst-level strict-accuracy 83.57 (PaLM 2 S 43.07, 55.76). §2.1: “one instruction limits the number of paragraphs to 5, whereas another instruction asks for less than 20 words”; illogical prompts were removed
- IFEval breakdown computed 2026-10-24 by running the IFEval code (google-research/instruction_following_eval, evaluation_lib strict) on GPT-4’s published responses (data/input_response_data_gpt4_20231107_145030.jsonl; one prompt whose wording was later edited matched by position); totals reproduce Table 3 (76.89, 83.57). All instructions followed by prompts with 1/2/3 instructions: 85.9% (305 prompts), 69.8% (179), 50.9% (57); each instruction followed: 85.9%, 82.7%, 81.3%. By instruction: letter_frequency 60.6% (33), no_comma 66.7% (66), number_sentences 67.3% (52), number_words 71.2% (52), response_language 96.8% (31), keywords existence 97.4% (39), title 100% (37), json_format 100% (17)
- Jaroslawicz, Whiting, Shah and Maamari, “How Many Instructions Can LLMs Follow at Once?” (IFScale; arXiv:2507.11538 v1, 2025-07-15, https://arxiv.org/abs/2507.11538, checked 2026-10-24): a business report that must include 10 to 500 given words, 20 models, 5 runs each. Appendix A Table 1, accuracy at 10/50/100/250/500: gemini-2.5-pro-preview 100.0/99.6/98.4/84.8/68.9, o3 (high) 100.0/99.6/98.2/97.8/62.8, gpt-4.1 98.0/98.8/95.4/74.0/48.9, claude-sonnet-4 100.0/98.0/94.4/77.2/42.9, gpt-4o 94.0/82.8/49.0/22.2/15.4. Primacy effect ratio (error rate in the final third ÷ the first third) at 250, all 23 rows: 1.48, 1.77, 1.49, 0.95, 1.15, 0.49, 1.55, 2.06, 1.67, 1.40, 1.15, 1.67, 2.89, 1.80, 1.31, 1.22, 1.60, 2.62, 2.33, 2.28, 1.50, 1.66, 1.68 (21 above 1). §4.6: “while packing more important instructions towards the beginning of a prompt may help, it becomes a less effective strategy once extreme densities are reached”
- Jiang et al., “FollowBench: A Multi-level Fine-grained Constraints Following Benchmark for Large Language Models” (arXiv:2310.20410 v3, 2024-06-05, https://arxiv.org/abs/2310.20410, checked 2026-10-24): Table 2, Hard Satisfaction Rate (all constraints of an instruction satisfied) at levels L1–L5 (1 to 5 constraints): GPT-4-Preview-1106 84.7, 75.6, 70.8, 73.9, 61.9; GPT-3.5-Turbo-1106 80.3, 68.0, 68.6, 61.1, 53.2
- The bottle request and reply, the bike-ad rules and lines, the mission prompt and the habit wording are illustrative, written for this lesson on 2026-10-24; the reply’s and the lines’ rule checks are computed by code in the lesson
- aifree.dev catalog, chatgpt-free (verified 2026-10-12, live check 2026-10-24): ChatGPT Free Plan
- Arithmetic in the text: 0.81 × 0.81 × 0.81 = 0.531, about 0.53 (0.81 is close to GPT-4’s 81.3% per instruction in three-instruction prompts above); five rules at 0.9 each: 0.9⁵ = 0.590, about 59%
Published Oct 24, 2026 · updated Oct 26, 2026