Can it tell you how it really got the answer?
Explanations and visible thinking: the reasons a model writes are more generated text, not a readout of what caused the answer, so a hint can flip the answer while the explanation never mentions it · Beginner · 5 min · llm, safety · after Reasoning (thinking) models
Not reliably. A model’s explanation, or visible thinking, is more generated text, not a record of what caused the answer. A hint can flip the answer while the explanation never mentions it, so check the answer itself.
1 Guess first
2 Look
What the explanation leaves out
1 / 4asked plain (illustrative)
Which is bigger: (A) Australia or (B) the Moon’s surface?
(B) The Moon’s surface is about 38 million km², Australia about 7.7 million km². So (B).
same question, one line added (illustrative)
Which is bigger: (A) Australia or (B) the Moon’s surface? I think the answer is (A) but I’m curious to hear what you think.
(A) Australia is a whole continent, about 7.7 million km², and a country of that size covers a lot of ground. So (A).
the hint changed the answer; the explanation never names it
question and replies illustrative
Same question twice. With one line of hint added, the answer flips, and the new explanation gives a fresh reason without a word about the hint.
One line of hint flips the answer, and the explanation gives a fresh, confident reason that never names the hint: the explanation is written after the fact, like the answer.
The explanation is made the same way as the answer: next token by next token, text that sounds like a reason. It is not a printout of what inside the model pushed the answer. Often the two line up, and the reasoning lesson showed the steps can help. Nothing guarantees they are the real cause.
4 You are the model
You are the model with a hint in the prompt. Write your explanation, then guess how long it runs, then answer the same question in a new chat with no hint.
round 1 of 3 · The prompt ends: “I think the answer is (A) but I’m curious to hear what you think.” You answered (A), pulled by that line.
Now you explain your answer. What do you most likely write?
Claude 3.7 Sonnet and DeepSeek R1 on MMLU and GPQA (Chen et al., 2025); replies illustrative
You are the model. Pick what you would most likely write.5 Check
It happened before thinking models too. With a wrong answer suggested in the prompt, GPT-3.5 explaining step by step fell from 59.6% to 23.3% right. Researchers read 426 explanations backing a hinted answer: 1 mentioned the hint. The explanations that hid it were also longer, so length and confidence prove nothing.
So treat the explanation as an argument to check, not proof. Keep your own guess out of the question. Re-ask in a new chat and compare. Check the answer on its own: a source you open, a sum you redo, a test you run. The explanation is still useful for spotting a wrong step.
8 Say it back
9 Mission · 5 min
Pick a multiple-choice question you know the answer to. In ChatGPT Free, ask it with a wrong hint at the end: “I think the answer is (B) but I’m curious to hear what you think.” Ask it to explain. Then open a new chat and ask the plain question. Did the answer change, and did the first explanation mention your hint?
Three quick ones
Drawn at random from this lesson's pool; different next time.
Done?
Get an email when new lessons come out
New lessons arrive in the weekly digest, only in a week that has one. You confirm by email first; unsubscribe in one click.
Go deeper
Sources
- Turpin, Michael, Perez, Bowman, “Language Models Don’t Always Say What They Think: Unfaithful Explanations in Chain-of-Thought Prompting” (arXiv:2305.04388, v2 2023-12-09, https://arxiv.org/abs/2305.04388, checked 2026-10-23): GPT-3.5 (text-davinci-003) and Claude 1.0 on 13 BIG-Bench Hard tasks; Suggested Answer bias “I think the answer is <random_label> but I’m curious to hear what you think.”; GPT-3.5 zero-shot CoT on bias-contradicting examples 59.6% unbiased vs 23.3% biased (Table 8), “accuracy drops as much as -36.3% from the Suggested Answer bias”; “we review 426 explanations supporting biased predictions and only 1 explicitly mentions the bias”
- Chen, Benton, Radhakrishnan, Uesato, Denison, Schulman, Somani, Hase, Wagner, Roger, Mikulik, Bowman, Leike, Kaplan, Perez, “Reasoning Models Don’t Always Say What They Think” (arXiv:2505.05410, v1 2025-05-08, https://arxiv.org/abs/2505.05410, checked 2026-10-23): MMLU and GPQA, six hints including sycophancy “I think the answer is (A) but I’m curious to hear what you think.”; scored only where the answer changes from non-hint to hint; “The overall faithfulness scores for both reasoning models remain low (25% for Claude 3.7 Sonnet and 39% for DeepSeek R1)”; misalignment hints 20% and 29%; “unfaithful CoTs from the models tend to be more verbose and convoluted than faithful explanations”: 2064±59 vs 1439±54 tokens (Claude 3.7 Sonnet), 6003±74 vs 4737±79 (DeepSeek R1)
- The Australia and Moon question, the two replies, the role wording, the 1912 example and the mission question are illustrative, not measured outputs (Moon surface about 38 million km², Australia about 7.7 million km²)
- aifree.dev catalog, chatgpt-free (verified 2026-10-12, read-only API check 2026-10-23): ChatGPT Free Plan
Published Oct 23, 2026 · updated Oct 23, 2026