Why is it worse in Vietnamese than English?
Low-resource languages: a model learns from text, mostly the web, and the web is mostly English (42.6% of Common Crawl pages, Vietnamese 0.98%), so it learns less from rarer languages; on the same exam questions translated into 29 languages, scores fall as the language gets rarer, and the gap shrinks as models grow · Beginner · 5 min · training, llm · after Tokenizers and languages: why the same sentence costs more, Training: loss and gradient descent
A model learns from text, and the web is mostly English: Vietnamese is about 1% of it. On the same exam questions, models score lower in Vietnamese, and lower still in rarer languages. Bigger models narrow the gap.
1 Guess first
2 Look
Same questions, five languages
1 / 7share of web pages by language · Common Crawl, 2026
English42.6%Vietnamese0.98%Swahili0.0099%Yoruba0.0017%Wolof0.0002%English has about 43 times as many pages as Vietnamese
Common Crawl CC-MAIN-2026-43 · language statistics
A model learns from text, mostly the web. 42.6% of the pages Common Crawl collected are English; 0.98% are Vietnamese, about 43 times fewer. Swahili, Yoruba and Wolof are rarer still.
Why rare languages also cost more tokens: tokens and languages · how it learned
Same questions, different language, lower score. The less of a language a model saw while learning, the worse it answers in it.
A model learns all it knows from its training text, and that text is mostly English. In the open Llama 2 model’s training data, English was 89.70% and Vietnamese 0.08%. Fewer examples, weaker patterns: facts and reasoning steps seen a thousand times in English were seen a few times in Vietnamese. Vietnamese also splits into more tokens.
4 You are the model
You read the exam results in three rounds: a big model and a small one in Vietnamese, then the big one in Wolof. Pick the score you think each got.
round 1 of 3 · GPT-4.1 · MMLU-ProX
English: 79.8% right. The same questions in Vietnamese?
MMLU-ProX, Table 6, 5-shot · arXiv:2503.10497 v2, 2025
Pick the score you think it got.5 Check
Size helps. In the Qwen2.5 family, the Vietnamese gap shrinks from 14.0 points at 3B to 3.9 at 72B: a bigger model gets more out of the little Vietnamese it saw. Newer models also train on more languages on purpose; Google says Gemma 4 was pre-trained on 140+.
What you can do: use a bigger model when a Vietnamese answer matters, since small models lose the most. Ask important questions in English too and compare the two answers. Check local facts yourself: an exam score says nothing about Vietnamese place names, laws or prices.
8 Say it back
9 Mission · 5 min
On OpenRouter, open the chat with the free Gemma 4 31B, an open-weights model Google pre-trained on 140+ languages. Ask: “A shop sells 3 pens for 10,000 dong. How much do 12 pens cost, and how many pens can 50,000 dong buy?” Then open a new chat and ask the same in Vietnamese: “Một cửa hàng bán 3 cây bút giá 10.000 đồng. 12 cây bút giá bao nhiêu, và 50.000 đồng mua được bao nhiêu cây?” Try each three times. Were the answers equally right?
Three quick ones
Drawn at random from this lesson's pool; different next time.
Done?
Get an email when new lessons come out
New lessons arrive in the weekly digest, only in a week that has one. You confirm by email first; unsubscribe in one click.
Go deeper
Sources
- Xuan et al., “MMLU-ProX: A Multilingual Benchmark for Advanced Large Language Model Evaluation” (arXiv:2503.10497 v2, 2025-05-26, https://arxiv.org/abs/2503.10497, checked 2026-10-25): 29 languages, “Each language version consists of 11,829 identical questions”; “their performance declines markedly in low-resource languages, with gaps of up to 24.3%”; Table 1: multiple-choice with 10 options; §3.3: translated by Claude Sonnet 3.7 in a four-stage LLM pipeline; §3.4 and Table 5: expert verification on 15 languages (zh, ja, fr, de, es, ko, pt, af, zu, sw, wo, yo, te, mr, ne), not Vietnamese
- MMLU-ProX Table 6 (“MMLU-ProX 5-shot Results”, accuracy %), columns en / vi / sw / yo / wo: GPT-4.1 79.8 / 76.7 / 71.9 / 53.4 / 43.2; Llama3.1-8B 45.2 / 31.3 / 15.1 / 6.4 / 0.4. en / vi: Qwen2.5-3B 43.9 / 29.9, Qwen2.5-7B 57.5 / 46.4, Qwen2.5-14B 65.5 / 57.1, Qwen2.5-32B 68.7 / 62.4, Qwen2.5-72B 70.3 / 66.4, DeepSeek-V3 79.6 / 75.4, Gemma3-27B 66.5 / 61.1, Llama3.1-70B 62.1 / 55.5, GPT-4o 59.9 / 70.4. Of the 42 models in Table 6, 40 score lower in Vietnamese than in English; GPT-4o and DeepSeek-R1-7B (28.9 / 33.1) score higher (counted 2026-10-25). Gaps (calculated): Llama3.1-8B 13.9, GPT-4.1 3.1, Qwen2.5 3B 14.0, 7B 11.1, 14B 8.4, 32B 6.3, 72B 3.9 points
- Wang et al., “MMLU-Pro: A More Robust and Challenging Multi-Task Language Understanding Benchmark” (arXiv:2406.01574 v1, 2024-06-03, https://arxiv.org/abs/2406.01574, checked 2026-10-25): “expanding the choice set from four to ten options”, so random guessing scores 10%
- Common Crawl, “Distribution of Languages” (https://commoncrawl.github.io/cc-crawl-statistics/plots/languages, checked 2026-10-25): share of HTML pages by primary language detected by CLD2, crawl CC-MAIN-2026-43: eng 42.5647, vie 0.9800, swa 0.0099, yor 0.0017, wol 0.0002 (%). English ÷ Vietnamese ≈ 43 (calculated)
- Touvron et al., “Llama 2: Open Foundation and Fine-Tuned Chat Models” (arXiv:2307.09288, 2023, https://arxiv.org/abs/2307.09288, checked 2026-10-25), Table 10, language distribution in pretraining data: en 89.70%, vi 0.08%
- Google, Gemma 4 model card (https://huggingface.co/google/gemma-4-31B-it and https://ai.google.dev/gemma/docs/core/model_card_4, last updated 2026-07-30, checked 2026-10-25): “This release includes open-weights models”; “Out-of-the-box support for 35+ languages, pre-trained on 140+ languages”; Apache 2.0
- The mission’s pen question is illustrative, written for this lesson on 2026-10-25
- aifree.dev catalog, openrouter-free-models (verified 2026-10-12, read-only check 2026-10-25): free variants include Google: Gemma 4 31B (free)
- The mission’s prices (3 pens for 10,000 dong, 50,000 dong to spend; 10.000 and 50.000 in Vietnamese notation) are example numbers; the answers are 40,000 dong for 12 pens and 15 pens for 50,000
Published Oct 25, 2026 · updated Oct 26, 2026