How do I know my new prompt is actually better?
Evals: a fixed question set, a grader and several runs · Builder · 5 min · sampling, llm · after Temperature and sampling
An eval is a fixed set of questions, a grader that scores each answer the same way, and several runs per prompt. A new prompt is better only if it wins on the same set by more than runs vary.
1 Guess first
2 Look
Same 10 questions, two prompts
1 / 4old prompt
France: Paris Italy: Rome Japan: Tokyo Germany:
new prompt
Capital cities: France: Paris Italy: Rome Japan: Tokyo Germany:
Answers to 10 capital questions, old prompt run 2, new prompt run 4 question old · run 2 new · run 4 Germany (Berlin) ✗ Frankfurt ✗ Frankfurt Spain (Madrid) ✓ Madrid ✓ Madrid Russia (Moscow) ✗ Anton ✓ Moscow China (Beijing) ✗ Chong ✓ Beijing Egypt (Cairo) ✗ Islamabad ✓ Cairo Greece (Athens) ✓ Athens ✓ Athens England (London) ✗ Edinburgh ✓ London Ireland (Dublin) ✗ Geneva ✗ Cork Austria (Vienna) ✗ Ger ✗ Tar Canada (Ottawa) ✗ Toronto ✗ G score 2 of 10 6 of 10 measured with GPT-2 small, temperature 1, 5 seeded runs, exact match · 2026-10-23
One run each, graded by exact match: the new prompt 6 of 10, the old one 2. Decisive?
One run is a dice roll: the same prompt on the same questions scores 3 one time and 6 the next.
An eval has three parts. A fixed set of questions, the same for every prompt. A grader: exact match when there is one right answer, a rubric for open answers, or an LLM-as-judge, which agreed with humans over 80% of the time in a 2023 study but is swayed by answer order and length. And several runs.
4 You are the model
You review a prompt change. For four real scoreboards from the picture’s runs, decide: ship the new prompt, not proven yet, or keep the old one.
you review the prompt change · 10 capital questions, exact match · what does each scoreboard prove?
scoreboard 1 of 4
2 questions (Germany, Spain), run 1: old prompt 0 of 2, new prompt 1 of 2.
measured with GPT-2 small, temperature 1, 5 seeded runs, exact match · 2026-10-23
You review the change. Ship it, wait for more runs, or keep the old prompt?5 Check
The next trap is overfitting. Tweak a prompt until it passes your 10 questions and it fits those 10, not the task. Keep held-out questions you never look at while writing, add real failures to the set as they come, and grow it: Anthropic advises more questions with automatic grading over fewer graded by hand.
7 Say it back
8 Mission · 5 min
With a free Gemini API key, build a tiny eval: 5 questions with one exact answer each, your old prompt and a new one, 3 runs each on gemini-3.5-flash-lite at temperature 1 (30 calls, a short script). Count exact matches per run. Did one prompt’s score change between runs? Is the gap between prompts bigger than that?
Three quick ones
Drawn at random from this lesson's pool; different next time.
Done?
Go deeper
Sources
- Measured with scripts/explain-data/evals.ts on 2026-10-23: GPT-2 small (Hugging Face gpt2, run locally), temperature 1, 5 seeded runs per prompt, exact match on the first sampled token. 10 test questions: plain sentence “The capital of X is” runs 0, 0, 0, 0, 1 (0.537); old prompt (France: Paris / Italy: Rome / Japan: Tokyo / X:) 3, 2, 2, 4, 3 (3.409); new prompt (“Capital cities:” + the old one) 4, 4, 3, 6, 3 (3.61). 6 held-out questions: old 1, 3, 1, 4, 5 (2.47); new 4, 3, 3, 4, 3 (2.764). Numbers in brackets: the long-run score, the sum of GPT-2’s odds for each right token
- https://arxiv.org/abs/2306.05685 (Zheng et al., Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena, v4 2023-12-24, checked 2026-10-23): “strong LLM judges like GPT-4 can match both controlled and crowdsourced human preferences well, achieving over 80% agreement, the same level of agreement between humans”; “position, verbosity, and self-enhancement biases”
- https://platform.claude.com/docs/en/test-and-evaluate/develop-tests (checked 2026-10-23): exact match “perfect for tasks with clear-cut, categorical answers”; string match, code-graded, LLM-graded; “Prioritize volume over quality: More questions with slightly lower signal automated grading is better than fewer questions with high-quality human hand-graded evals”
- https://ai.google.dev/gemini-api/docs/pricing (updated 2026-10-22, checked 2026-10-23): Gemini 3.5 Flash-Lite free tier, input and output “Free of charge”; https://ai.google.dev/gemini-api/docs/rate-limits (updated 2026-10-15): “Rate limits are applied per project, not per API key”; aifree.dev catalog google-gemini-developer-api-free-tier (verified 2026-10-12)
Published Oct 23, 2026 · updated Oct 23, 2026