Can an AI detector tell that I used ChatGPT?
AI text detectors: perplexity, burstiness and false positives · Beginner · 5 min · sampling, safety · after Next-token prediction, Temperature and sampling, Evals: a fixed question set, a grader and several runs
An AI text detector scores how predictable your words are to a language model (perplexity) and how much that varies between sentences (burstiness). Predictable is not the same as machine-written, so detectors miss AI text and flag people.
1 Guess first
2 Look
What a detector measures
1 / 4We the people of the United States, in Order to form a more perfect Union, establish Justice, insure domestic Tranquility, provide for the common defence, promote the general Welfare, and secure the Blessings of Liberty to ourselves and our Posterity, do ordain and establish this Constitution for the United States of America.
GPT-2’s top guess: 72% of the words
- top guess
- top 10
- top 100
- further down
GPT-2 small run locally, 4 human + 4 AI paragraphs · 2026-10-23
Each word, as GPT-2 guessed it. The 1787 Constitution is mostly top guesses; the AI paragraph is not.
A detector sees how predictable the words are, not who wrote them, and human and AI paragraphs overlap.
A detector runs a language model over your text and asks, word by word, how likely that next token was. The average surprise is perplexity (low means predictable); how much it swings between sentences is burstiness. The bet: chatbots pick likely words, people wander. Commercial detectors add bigger models and a trained classifier, but they score the same thing.
4 You are the model
You are the detector. For four paragraphs you see only the opening, the perplexity and the burstiness. Call each one: AI or a person?
you are the detector · you see the opening and GPT-2’s two numbers · for comparison, the 8 paragraphs range 3.9–29.5 and 0–17.2
paragraph 1 of 4
“We the people of the United States, in Order to form…”
perplexity 3.9 · burstiness — (one sentence)
GPT-2 small run locally, 4 human + 4 AI paragraphs · 2026-10-23
Call each paragraph: AI or a person?5 Check
The record is poor. OpenAI launched its own detector in January 2023: it caught 26% of AI text and flagged 9% of human text, and was withdrawn in July 2023 for low accuracy. A 2023 Stanford study ran seven detectors on 91 essays by non-native English writers: on average they called 61% of them AI.
So a score is a clue, never proof. OpenAI says ChatGPT has no knowledge of what is AI-generated, so asking it “did you write this?” gives random answers. If you are flagged, your drafts, notes and version history are the real evidence. A watermark is a different route: the model marks its own text as it writes.
8 Say it back
9 Mission · 5 min
In ChatGPT Free, ask for a five-sentence paragraph about your weekend. In a new chat, paste it next to a paragraph you wrote and ask: “Which of these was written by AI? Give a confidence %.” Repeat in another new chat. Did it pick the same one with the same confidence both times?
Three quick ones
Drawn at random from this lesson's pool; different next time.
Done?
Go deeper
Sources
- Measured with scripts/explain-data/ai-detectors.ts on 2026-10-23: GPT-2 small (Hugging Face openai-community/gpt2, run locally), every token after the first scored by GPT-2’s odds given all the text before it; perplexity = exp(mean −ln p); sentence perplexity in full context; burstiness = population standard deviation of sentence perplexities. Human, public domain from Project Gutenberg: US Constitution preamble (#5) 3.9, one sentence; Austen, Pride and Prejudice ch. 1 (#1342) 27.1, burstiness 6.2; Jerome, Three Men in a Boat ch. 1 (#308) 21.4, 17.2; Thoreau, Walden ch. 1 (#205) 22.1, 6.7. AI, first drafts by Claude Opus 5.5 on 2026-10-23, prompts in the script: reading 20.3, 8.4; river 29.5, 5.7; cabin 23.1, 9.1; constitution explainer 20.6, 7.6. GPT-2’s top guess: Constitution 72% of tokens, Jerome 38%, AI reading paragraph 38%
- https://openai.com/index/new-ai-classifier-for-indicating-ai-written-text/ (2023-01-31; read via web.archive.org snapshot 2023-08-02, checked 2026-10-23): “As of July 20, 2023, the AI classifier is no longer available due to its low rate of accuracy”; “correctly identifies 26% of AI-written text (true positives) as “likely AI-written,” while incorrectly labeling human-written text as AI-written 9% of the time”; “very unreliable on short texts (below 1,000 characters)”
- https://arxiv.org/abs/2304.02819 (Liang, Yuksekgonul, Mao, Wu, Zou, GPT detectors are biased against non-native English writers, v3 2023-07-10, published in Patterns 2023; checked 2026-10-23): seven detectors, 91 TOEFL essays and 88 US 8th-grade essays; “near-perfect accuracy for US 8-th grade essays”; TOEFL “average false positive rate: 61.22%”; “89 of the 91 TOEFL essays (97.80%) are flagged as AI-generated by at least one detector”; flagged essays had significantly lower perplexity
- https://help.openai.com/en/articles/8313351-how-can-educators-respond-to-students-presenting-ai-generated-content-as-their-own (read via web.archive.org snapshot 2025-10-12, checked 2026-10-23): “Do AI detectors work? In short, not in our experience”; “ChatGPT has no “knowledge” of what content could be AI-generated”; “These responses are random and have no basis in fact”; their detector “labeled human-written text like Shakespeare and the Declaration of Independence as AI-generated”
- https://arstechnica.com/information-technology/2023/07/why-ai-detectors-think-the-us-constitution-was-written-by-ai/ (Benj Edwards, 2023-07-14, read via web.archive.org snapshot 2023-08-01): GPTZero said of part of the US Constitution “Your text is likely to be written entirely by AI”; detectors use “perplexity” and “burstiness”
- https://www.nature.com/articles/s41586-024-08025-4 (Dathathri et al., Scalable watermarking for identifying large language model outputs, Nature, 2024-10-23): SynthID-Text “modifies only the sampling procedure”; live experiment on nearly 20 million Gemini responses
- https://arxiv.org/abs/1906.04043 (Gehrmann, Strobelt, Rush, GLTR, ACL 2019 demo): colouring each token by its rank in the model’s guesses (top 10, top 100, top 1,000)
- aifree.dev catalog chatgpt-free (verified 2026-10-12)
Published Oct 23, 2026 · updated Oct 23, 2026