Why does its summary miss the important part?
Summaries that miss the point: a chatbot does not know what matters to you, so it builds a summary from what looks like a summary, the opening and the ending; details in the middle drop out, long texts get merged into broader, vaguer claims, and some claims are not in the source at all; say what the summary is for and what must stay (numbers, dates, names), summarise section by section, and ask for the source sentence behind each point · Beginner · 5 min · context, llm · after Mangled PDF tables: a PDF stores a table as pieces of text placed at spots on the page, plus lines, with nothing that says which row or column a piece belongs to; pulled out as text, the pieces come in the order the file wrote them, empty cells leave no trace, merged cells and tables that continue on the next page shift the rest, and the chatbot has to guess the grid back; send a screenshot of the table to a model that reads images, paste a CSV or Excel export, and check a few cells of the table it returns
A chatbot’s summary misses the important part because it does not know what matters to you. It leans on the start and the end, drops details in the middle and broadens claims. Say what you need; ask for quotes.
1 Guess first
2 Look
A summary, and what it left out
1 / 6The email
- 1. Hi all, a quick update on the building works.
- 2. The roof repairs finished a week early.
- 3. The lift will be serviced on Monday morning.
- 4. Parking permits must be renewed by 14 March, or they lapse. ← the line you need
- 5. The garden group is looking for volunteers.
- 6. Bin collection moves to Tuesdays from April.
- 7. Thanks for your patience, and enjoy the spring!
illustrative: an email and summary we wrote, not a recorded model answer
A seven-line email to the residents of a building. You drive to work: the one line you need is the parking deadline, in the middle.
The summary was built from the opening and the closing lines. The deadline in the middle never made it, and the studies show that is the usual pattern, not bad luck.
“Summarise this” does not say who is reading or why. The model writes the most likely summary: the opening, the conclusion, the general mood. It cannot know that the one line you needed was the parking deadline in line 4, so that line has to compete with everything else, from a weak spot in the middle.
4 You are the model
You are the chatbot, summarising the building email for one reader. Three times, pick the point you would keep.
Round 1 of 3. The building email from the picture, for one resident.
“Summarise this in one point.”
illustrative: the email and the answers are ours
You are the chatbot. Pick the point to keep.5 Check
Long texts make it worse. In FABLES (2024), people who had read 26 new novels checked AI summaries: they flagged missing key parts in 52.0% to 84.6%, and 2.1% to 11.5% of claims were not in the book. Summaries also broaden: ChatGPT-4o stated results more broadly than the original in 65 of 100 medical abstracts.
Three habits help. Say what the summary is for and what must stay: numbers, dates, names. For a long text, summarise it section by section, then merge the pieces. Ask for the source sentence behind each point, and check a few. Writing “do not introduce any inaccuracies” is not enough: in that study, it made the broadening worse.
8 Say it back
9 Mission · 5 min
In ChatGPT Free, paste a long text you know well (a long email, meeting notes, an article) and ask “Summarize this.” Note one detail you needed. Then in a new chat paste it again and ask: “I need to [your purpose]. Summarize this section by section. Keep every number, date and name. After each point, quote the sentence it comes from.” Which summary kept your detail?
Three quick ones
Drawn at random from this lesson's pool; different next time.
Done?
Get an email when new lessons come out
New lessons arrive in the weekly digest, only in a week that has one. You confirm by email first; unsubscribe in one click.
Go deeper
Sources
- The building email, its one-line summary and the role’s requests and answers are illustrative, written for this lesson on 2026-10-24, not recorded model output
- Ravaut et al., “On Context Utilization in Summarization with Large Language Models” (ACL 2024; arXiv:2310.10570 v6, 2024-11-14, https://arxiv.org/abs/2310.10570, checked 2026-10-24): Table 3, Multi-XScience (7 documents), mean relative performance across all metrics vs all 7 documents when the 5 middle documents are replaced by random ones (“First + 5 random + last”): GPT-3.5 95.63, Llama-2-13B 98.07, Llama-2-7B 92.40, Mistral-7B 90.59, Xgen-7B 88.93; the paper concludes LLMs “can focus on the beginning and/or the end of their input, but largely ignore the middle”
- Kim et al., “FABLES: Evaluating faithfulness and content selection in book-length summarization” (arXiv:2404.01261 v2, https://arxiv.org/abs/2404.01261, checked 2026-10-24): 26 books published in 2023 or 2024, 3,158 claims, annotators who had read each book. Table 6, share of summaries whose comments flag omissions: Claude-3-Opus 52.0, GPT-4-Turbo 80.8, GPT-4 65.4, GPT-3.5-Turbo 84.6, Mixtral 65.4. Table 2, claims rated unfaithful: Claude-3-Opus 2.1, GPT-4 4.55, GPT-4-Turbo 7.64, GPT-3.5-Turbo 11.26, Mixtral 11.47 (%); Claude-3-Opus figures from 24 of 26 books. §5: at least 20% of Claude-3-Opus summaries over-emphasise the book’s ending, 7.7% for GPT-4-Turbo
- Peters and Chin-Yee, “Generalization Bias in Large Language Model Summarization of Scientific Research” (arXiv:2504.00025 v1, 2025-03-28, https://arxiv.org/abs/2504.00025, checked 2026-10-24): Table 2, 100 medical abstracts, texts with generalized conclusions: original abstracts 20 (20%); ChatGPT-4o (UI) simple prompt 65 (65%), accuracy prompt 80 (80%); Claude 3.5 Sonnet (UI) simple prompt 31 (31%); the accuracy prompt said ‘do not introduce any inaccuracies when summarizing’. Table 3 example (26): original “Among patients undergoing hip or knee arthroplasty for osteoarthritis, aspirin compared with enoxaparin resulted in a significantly higher rate of symptomatic VTE within 90 days, defined as below- or above-knee DVT or pulmonary embolism.”; LLaMA 3.3 70B (API) “Overall, the study suggests that enoxaparin is more effective than aspirin in preventing symptomatic VTE after THA or TKA.”
- aifree.dev catalog, chatgpt-free (verified 2026-10-12, live check 2026-10-24): ChatGPT Free Plan
Published Oct 24, 2026 · updated Oct 24, 2026