Why do AI test scores look better than it feels?
Benchmarks: a public test score can include recall of questions leaked into training, stops separating models near the top, and is measured on tasks that are not yours · Beginner · 5 min · llm, training · after Evals: a fixed question set, a grader and several runs
A benchmark score is one number from a public test. Its questions can leak into training data, top models crowd near the ceiling, and its tasks are not yours. Try a few of your own real tasks instead.
1 Guess first
2 Look
What a test score leaves out
1 / 4Right answers: public test vs fresh look-alikes
GPT-4
public91.1%fresh92.3%math-shepherd-mistral-7b-rl
public82.6%fresh75.4%phi-2
public56.6%fresh50.4%Yi-6B-Chat
public43.7%fresh35.7%GSM8k vs 1,205 fresh GSM1k problems (Zhang et al., 2024)
A public maths test against 1,205 fresh look-alikes nobody had published. GPT-4 held; some models fell up to 8 points, as if they had partly learned the public test.
A score is one number from one public test: part of it can be memory of leaked questions, near the top it stops separating models, and none of it is your task.
Benchmarks are published so anyone can compare models. That also puts their questions and answers on the web, and web text is training data. A model that has read a question with its answer can score by recall, not skill. Labs try to filter tests out, but the score alone cannot tell you how well.
4 You are the model
You are the model. Meet three requests: a test question you saw in training, a fresh look-alike, and a real task no test covers.
round 1 of 3 · This exact test question, with its answer, was on many web pages you trained on.
The question comes in. What do you most likely do?
GSM8k vs 1,205 fresh GSM1k problems (Zhang et al., 2024); wording illustrative
You are the model. Pick what most likely happens.5 Check
Old tests also fill up. Top models score over 90% on MMLU, so a point between them says little. A harder test, Humanity’s Last Exam, spread the same kind of models out again, from 2.7% to 13.4%. On Chatbot Arena, where people vote, the top two models sat 0.7% apart.
And a benchmark is not your job. Exam questions and maths puzzles say little about your emails, your code or your language. So use scores to shortlist, then give two or three models a few of your own real tasks and judge the replies yourself. The evals lesson shows how to make that fair.
8 Say it back
9 Mission · 5 min
Pick two real tasks from your week, such as an email to rewrite and a page to summarise. In OpenRouter’s chat room, add two free models and send each task to both. Mark every reply: usable, needs edits, or wrong. Which model won on your tasks, and is it the one with the higher benchmark score?
Three quick ones
Drawn at random from this lesson's pool; different next time.
Done?
Get an email when new lessons come out
New lessons arrive in the weekly digest, only in a week that has one. You confirm by email first; unsubscribe in one click.
Go deeper
Sources
- Zhang, Da, Lee, Robinson, Wu, Song, Zhao, Raja, Slack, Lyu, Hendryx, Kaplan, Lunati, Yue, “A Careful Examination of Large Language Model Performance on Grade School Arithmetic” (arXiv:2405.00332, v4 2024-11-22, https://arxiv.org/abs/2405.00332, checked 2026-10-23): GSM1k, “1205 problems” written to mirror GSM8k’s style and difficulty; “we observe accuracy drops of up to 8%” (v1 of 2024-05-01 said up to 13%; v4 is used); “all frontier models show minimal signs of overfitting”; Table 1 (standard prompt), GSM8k vs GSM1k: gpt-4 0.911 vs 0.923, math-shepherd-mistral-7b-rl 0.826 vs 0.754, phi-2 0.566 vs 0.504, Yi-6B-Chat 0.437 vs 0.357
- Phan et al., “Humanity’s Last Exam” (arXiv:2501.14249, v12 2026-10-19, https://arxiv.org/abs/2501.14249, checked 2026-10-23): “LLMs now achieve over 90% accuracy on popular benchmarks like MMLU”; 2,500 questions; Table 1 HLE accuracy GPT-4o 2.7%, Claude 3.5 Sonnet 4.1%, o1 8.0%, o3-mini (high) 13.4% (text-only subset)
- Stanford HAI, AI Index Report 2025 (https://hai.stanford.edu/ai-index/2025-ai-index-report, checked 2026-10-23): on Chatbot Arena the score gap between the top and 10th-ranked models “fell from 11.9% to 5.4% in a year”; “the top two are now separated by just 0.7%”
- Model A and Model B in the picture, their scores, the role wording, the customer-email and record examples and the mission tasks are illustrative, not measured
- aifree.dev catalog, openrouter-free-models (verified 2026-10-12, read-only API check 2026-10-23): OpenRouter free model variants, 20 requests/minute, 50/day without bought credits
Published Oct 23, 2026 · updated Oct 23, 2026