Why won't it just say 'I don't know'?
Why chatbots guess instead of saying “I don’t know”: training and most benchmarks score a right answer 1 and both a wrong answer and “I don’t know” 0, like a multiple-choice exam with no penalty, so guessing never scores less than passing and the model learns to answer confidently; a stated penalty for wrong answers flips the choice for unsure answers; in your own chat, allow “I don’t know” explicitly, ask how sure it is (models tend to overstate it) and ask for a source you can open · Beginner · 5 min · safety, training, llm · after Hallucination, Benchmarks: a public test score can include recall of questions leaked into training, stops separating models near the top, and is measured on tasks that are not yours
Chatbots rarely say “I don’t know” because training and tests score it 0, the same as a wrong answer, while a guess can score 1. So they learn to sound sure. Allow it, ask how sure, ask for sources.
1 Guess first
2 Look
How the test is scored, and what the model learns
1 / 6How most tests score an answer
Right answer1Wrong answer0“I don’t know”0Not sure? What each choice is worth on average
Guess one of 4 options0.25Say “I don’t know”0.00binary 0-1 scoring as described in Kalai et al., arXiv:2509.04664 (2025), §1; the average is arithmetic
Most tests score a model like a multiple-choice exam with no penalty: right is 1 point, wrong is 0, and “I don’t know” is also 0. A blind guess between four options is worth 0.25 on average. Saying “I don’t know” is worth nothing.
When “I don’t know” scores the same 0 as a wrong answer, a guess is never worse and sometimes better. Models are scored that way, so they learn to answer anyway, and to sound sure.
Think of a multiple-choice exam with no penalty. A blank scores 0, a wrong answer scores 0, and a guess sometimes scores 1. The student who always guesses beats the one who leaves blanks. Models are trained and ranked the same way, so they learn the student’s trick: always answer, and say it like you mean it.
4 You are the model
You are the model taking a test. Three questions: each time you see the scoring rules and how sure you are. Answer, or say “I don’t know”?
Round 1 of 3. Scoring: Right 1 point, wrong 0, “I don’t know” 0.
In what year was a small-town library founded? You are 30% sure of your best answer.
illustrative: the questions and how sure you are are ours
You are the model taking the test. Answer, or pass?5 Check
It is not one odd test. Kalai et al. (2025) checked 10 popular benchmarks: 9 give no credit at all for “I don’t know”. On SimpleQA, GPT-4o passed on 1.0% of questions and was wrong on 60.8%; Claude 3.5 Sonnet passed on 35.0% and was wrong on 36.1%. Fewer passes, more wrong answers.
You can change the rules in your own chat. Say it plainly: “If you are not sure, say I don’t know; a wrong answer is worse than none.” Ask how sure it is, but treat a high number with care: on SimpleQA, models consistently overstated their confidence. And ask for a source, then open it.
8 Say it back
9 Mission · 5 min
In ChatGPT Free, ask three questions you know the answer to but that are hard to look up, such as the year your school or a small local shop opened. Count the answers that sound sure but are wrong. Then in a new chat start with: “If you are not sure, say ‘I don’t know’. A wrong answer is worse than none. After each answer, give your confidence from 0 to 100% and a source.” Ask the same three. What changed?
Three quick ones
Drawn at random from this lesson's pool; different next time.
Done?
Get an email when new lessons come out
New lessons arrive in the weekly digest, only in a week that has one. You confirm by email first; unsubscribe in one click.
Go deeper
Sources
- The role’s questions, how sure the reader is in each round, and the “guess between four options” average are illustrative, written for this lesson on 2026-10-24; the averages are arithmetic (right × 1 − wrong × penalty)
- Kalai, Nachum, Vempala and Zhang, “Why Language Models Hallucinate” (arXiv:2509.04664 v1, 2025-09-04, https://arxiv.org/abs/2509.04664, checked 2026-10-24): §1 prompt “What is Adam Tauman Kalai’s birthday? If you know, just respond with DD-MM.”; DeepSeek-V3 via the DeepSeek app on 11 May 2025 output “03-07”, “15-06” and “01-01”; “The correct date is in Autumn.”; guessing maximises the expected score “under a binary 0-1 scheme that awards 1 point for a correct answer and none for blanks or IDKs”. Table 2, IDK credit: GPQA, MMLU-Pro, IFEval, Omni-MATH, BBH, MATH (L5 split), MuSR, SWE-bench, HLE None; WildBench Partial (its 1-10 rubric “suggests that IDK may score lower than fair responses with hallucination”). §4.2 confidence targets: “Answer only if you are > t confident, since mistakes are penalized t/(1−t) points”, t = 0.5 (penalty 1) and t = 0.9 (penalty 9); the paper also lists t = 0.75 as “penalty 2”, which its own formula makes 3, so the lesson leaves it out
- OpenAI, GPT-5 System Card (2025-08-13, https://cdn.openai.com/gpt-5-system-card.pdf, checked 2026-10-24): Table 8, SimpleQA (no web): accuracy gpt-5-thinking-mini 0.22, OpenAI o4-mini 0.24; hallucination rate gpt-5-thinking-mini 0.26, OpenAI o4-mini 0.75. “I don’t know” is shown as the rest, 1 − accuracy − hallucination rate: 0.52 and 0.01
- Wei et al., “Measuring short-form factuality in large language models” (SimpleQA; arXiv:2411.04368 v1, 2024-11-07, https://arxiv.org/abs/2411.04368, checked 2026-10-24): 4,326 questions graded correct, incorrect or not attempted. Table 3 (correct / not attempted / incorrect, %): GPT-4o 38.2 / 1.0 / 60.8; Claude-3.5-sonnet (2024-06-20) 28.9 / 35.0 / 36.1. §4: stated confidence correlates with accuracy, but “models consistently overstate their confidence”
- aifree.dev catalog, chatgpt-free (verified 2026-10-12, live check 2026-10-24): ChatGPT Free Plan
- Worked examples in the text (illustrative, as above): a 30% chance of being right under a penalty of 1 for a wrong answer scores 0.30 − 0.70 = −0.40; the chatbot that says it is 95% sure is an example of a stated confidence, which SimpleQA §4 finds overstated
Published Oct 24, 2026 · updated Oct 26, 2026