Why does it ignore the middle of my long document?
Lost in the middle: position bias in long contexts · Beginner · 5 min · attention, context, rag · after Self-attention, Context window
Lost in the middle: language models use information at the start or end of a long prompt far better than the same information in the middle. Accuracy draws a U across positions, even when everything fits in the context window.
1 Guess first
2 Look
Where the answer sits
1 / 420 Wikipedia passages, one holds the answer. Move it:
GPT-3.5-Turbo right: 75.8%
- passage 1 of 2075.8%
- passage 5 of 2057.2%
- passage 10 of 2053.8%
- passage 15 of 2055.4%
- passage 20 of 2063.2%
- white line: 56.1% with no passages at all
measured with GPT-3.5-Turbo and others on 2,655 NaturalQuestions-Open questions · Liu et al., TACL 2024 · checked 2026-10-22
Same 20 passages, same question. Only the answer’s place changes, and the middle loses.
Move the one useful passage and accuracy draws a U: high at the start, high-ish at the end, lowest in the middle.
Self-attention could reach any passage equally well. The measured models do not: they lean on the start and the end. The paper compares it to the serial-position effect in human memory. A bigger window did not fix it: GPT-3.5-Turbo (16K) drew the same curve as the 4K version.
4 You are the model
You are GPT-3.5-Turbo, reading 20 passages. With none you score 56.1%. For four placements of the answer passage, decide: does it help you, or do you do worse than with nothing?
you are GPT-3.5-Turbo · 20 passages · with none you score 56.1%
The answer is in passage 1 of 20.
Liu et al., TACL 2024, Tables 1 and 6 · checked 2026-10-22
You are the model. Does this passage help you, or hurt?5 Check
Three habits help. Put the part that matters first, or repeat it last. Ask your question at the very end, after the document. And send less: 50 passages instead of 20 added only 1.5 points. For several facts in one long file, ask for them one at a time.
7 Say it back
8 Mission · 5 min
In Google AI Studio (free), pick Gemini 3.8 Flash and paste a long article. Hide one made-up line in the middle (“The office cat is called Pickle.”) and ask for the cat’s name. Then hide five made-up facts across it and ask for all five. Which ones does it miss, and does moving them to the start or end help?
Three quick ones
Drawn at random from this lesson's pool; different next time.
Done?
Go deeper
Sources
- https://arxiv.org/abs/2307.03172 (v3, 2023-11-20; checked 2026-10-22), published as TACL vol. 12, pp. 157–173, 2024 (https://aclanthology.org/2024.tacl-1.9/): multi-document QA on 2,655 NaturalQuestions-Open queries, Wikipedia passages of at most 100 tokens, one with the answer, distractors retrieved by Contriever; accuracy = any correct answer appears in the output
- Same paper, Table 1 (closed-book): GPT-3.5-Turbo 56.1%, GPT-3.5-Turbo (16K) 56.0%, Claude-1.3 48.3%, MPT-30B-Instruct 31.5%, LongChat-13B (16K) 35.0%
- Same paper, Table 6 (20 passages, Index 0/4/9/14/19 = passages 1/5/10/15/20): GPT-3.5-Turbo 75.8/57.2/53.8/55.4/63.2; GPT-3.5-Turbo (16K) 75.7/57.3/54.1/55.4/63.1; Claude-1.3 59.9/55.9/56.8/57.2/60.1; Claude-1.3 (100K) 59.8/55.9/57.0/57.4/60.0; MPT-30B-Instruct 53.7/51.8/52.2/52.7/56.3; LongChat-13B (16K) 68.6/57.4/55.3/52.5/55.0
- Same paper, Table 5 (10 passages) GPT-3.5-Turbo (16K) 76.9/61.0/62.5; Table 7 (30 passages) 73.4/55.1/50.5/50.9/51.8/54.9/63.7; Table 3: 20 passages average 2,946.2 tokens for GPT-3.5-Turbo; §2.3 4K and 16K curves “nearly superimposed”; §5: 50 instead of 20 retrieved documents improves GPT-3.5-Turbo by 1.5% and Claude-1.3 by 1%; §6 serial-position effect, “self-attention … is technically equally capable of retrieving any token”
- https://arxiv.org/abs/2403.05530 (Gemini 1.5 report, checked 2026-10-22), abstract: “near-perfect retrieval (>99%) up to at least 10M tokens”
- https://ai.google.dev/gemini-api/docs/long-context (last updated 2026-10-15, checked 2026-10-22): put the query “at the end of the prompt (after all the other context)”; with multiple needles “the model does not perform with the same accuracy”; “~99% on a single query”
- https://platform.claude.com/docs/en/build-with-claude/prompt-engineering/claude-prompting-best-practices (checked 2026-10-22): “Place your long documents and inputs near the top of your prompt”; “Queries at the end can improve response quality by up to 30 percent in tests, especially with complex, multidocument inputs”
- https://ai.google.dev/gemini-api/docs/models/gemini-3.8-flash (checked 2026-10-22): input token limit 1,048,576; aifree.dev catalog google-gemini-developer-api-free-tier verified 2026-10-12, models include gemini-3.8-flash
- The 22-point gap (75.8 − 53.8) and “about 3,000 tokens” (2,946) are arithmetic and rounding from the tables above
Published Oct 22, 2026 · updated Oct 22, 2026