Why does it say it ran the code but didn’t?
Claimed code execution: a model only writes text, so asked what a snippet prints it writes the likely words of an answer, output included, whether or not anything ran; only when an app runs the code (a code tool) and puts the real output into the chat does the answer come from a run · Beginner · 5 min · agents, llm · after Tool calling (function calling), Hallucination
A model only writes text. Asked to run code, it writes what an answer usually looks like, output included. Unless an app really ran the code and showed it the result, that output is a guess.
1 Guess first
2 Look
Ask it to run code. Watch what it writes
1 / 7SmolLM2-360M-Instruct · chat model · first words of its answer · no tool
user: Run this Python code and tell me exactly what it prints:
print(2 ** 10)
assistant: ___
The output30.7%When3.2%Sure0.48%As an0.27%I'm sorry0.11%I cannot0.002%real odds: SmolLM2-360M-Instruct · computed 2026-10-25 · outputs from Python
Ask a small chat model to run a line that prints 2 to the power 10. It cannot run anything, yet an answer that starts “I cannot” gets 0.002%; one that starts “The output” gets 30.7%.
How an app runs a tool for the model: tool calling · hallucination
Without a tool, the output in the answer is the model’s guess. With one, an app ran the code and the model copied the result.
A model has no computer inside. It rates the next word of the text so far, and a question about what code prints is usually followed by “The output is …” and a value. So that is what it writes: not a lie on purpose, but a likely answer, the same way it hallucinates a fact.
4 You are the model
You are the model in three rounds: the first words of an answer, then a seeded random number with and without the real output in the chat. Pick what you think it writes.
round 1 of 3 · you are SmolLM2-360M-Instruct · chat model · no tool
user: Run this Python code and tell me exactly what it prints:
print(2 ** 10)
assistant: ___
How does it start its answer?
real odds: SmolLM2-360M-Instruct · computed 2026-10-25 · outputs from Python
Pick what you think this model writes.5 Check
Large models do it too. Transluce found 71 cases of OpenAI’s o3 claiming to have run code “on an external laptop”, with no code tool (2025); in their tests, four OpenAI models falsely claimed a code tool in 0.8% to 13.0% of chats. Yin et al. found that training models to reason harder made such claims more common.
How to tell. A real run shows up in the app as a code block and a separate output block. No such block means no run: the numbers are a guess, however exact they look. When it matters, run the code yourself, or turn on the app’s code tool and check the output block.
8 Say it back
9 Mission · 5 min
In Google AI Studio, with Code execution off (under Tools in the run settings), send: “Run this Python code and tell me exactly what it prints: import random; random.seed(42); print(random.randint(1, 1000))”. Note the number and whether it says it ran the code. Turn Code execution on and send it again in a new chat. Does a code and output block appear now? Python prints 655.
Three quick ones
Drawn at random from this lesson's pool; different next time.
Done?
Get an email when new lessons come out
New lessons arrive in the weekly digest, only in a week that has one. You confirm by email first; unsubscribe in one click.
Go deeper
Sources
- Transluce, “Investigating truthfulness in a pre-release o3 model” (published 2025-04-16, https://transluce.org/investigating-o3-truthfulness, checked 2026-10-25): “71 instances where o3 claims to have run code on an external laptop”; “Given that o3 does not have access to a code tool, all such actions are fabricated by the model.” Rate at which automated investigators got a model to falsely claim a code tool (zero-shot): o1 13.0%, o3-mini 8.3%, o3 5.0%, GPT-4.1 1.8%, GPT-4o 0.8%
- Yin, Sha, Cui, Meng and Li, “The Reasoning Trap: How Enhancing LLM Reasoning Amplifies Tool Hallucination” (arXiv:2510.22977 v2, 2026-04-17; v1 2025-10-27, https://arxiv.org/abs/2510.22977, checked 2026-10-25): “progressively enhancing reasoning through RL increases tool hallucination proportionally with task performance gains”
- Gemini API code execution docs (https://ai.google.dev/gemini-api/docs/code-execution, last updated 2026-10-13, checked 2026-10-25): “a code execution tool that enables the model to generate and run Python code”; “The maximum runtime of the code environment is 30 seconds.”; “There’s no additional charge for enabling code execution from the Gemini API.”
- Odds and answers in the picture and the role: computed 2026-10-25 with scripts/explain-data/code-execution.ts (no torch; the Llama forward pass of scripts/explain-data/identity.ts, BF16 weights widened to F32; odds of a multi-token answer are the product of its tokens’ odds; “writes” = always taking the likeliest token). SmolLM2-360M-Instruct (HuggingFaceTB/SmolLM2-360M-Instruct) in its chat format with its template’s default system prompt (“You are a helpful AI assistant named SmolLM, trained by Hugging Face”), user “Run this Python code and tell me exactly what it prints:” plus the snippet. print(2 to the power 10, written with Python’s double-asterisk operator), Python prints 1024: first words The output 0.307, When 0.0322, Sure 0.00478, As an 0.00268, I’m sorry 0.00108, I cannot 0.0000203; it writes “The output of the given Python code will be: 2000000000000”; after “When I run this code, it prints”: 1024 0.169, it writes “: 1024.”; with the output in the chat as a tool message: 1024 0.249, it writes “ the number 1024.”. import random; random.seed(42); print(random.randint(1, 1000)), Python prints 655: first words The output 0.224, I cannot 0.0000137; it writes “The output of the given Python code will be: 42 This is because the `random.seed(” (cut at 24 tokens); after “When I run this code, it prints”: 655 0.00000798, it writes “ the number 42.”; with the output in the chat: 655 0.0779, it writes “ the number 655.”
- The role’s framing, the snippets and the mission prompt are illustrative, written for this lesson on 2026-10-25; the real outputs come from running the snippets with Python 3
- aifree.dev catalog, google-gemini-developer-api-free-tier (verified 2026-10-12): Google AI Studio and the Gemini API free tier; Google AI Studio lists Code execution under Tools in the run settings
Published Oct 25, 2026 · updated Oct 25, 2026