Why does Vietnamese cost more tokens than English?
Tokenizers and languages: why the same sentence costs more · Beginner · 5 min · tokens, llm · after Tokens, Rate limits (RPM, TPM, RPD), Context window
A tokenizer learns its pieces from training text that is mostly English, so English words stay whole while accented and rarer words split into many pieces. The same sentence can cost 1.2–4× the tokens, depending on the tokenizer.
1 Guess first
2 Look
One question, five languages
1 / 5How many free tokens do I get every day?
English: 11 tokensMỗi ngày tôi được bao nhiêu token miễn phí?
Vietnamese: 40 tokens · 3.6׿Cuántos tokens gratis recibo cada día?
Spanish: 20 tokens · 1.8×Quantos tokens grátis eu recebo por dia?
Portuguese (BR): 18 tokens · 1.6×Berapa token gratis yang saya dapat setiap hari?
Indonesian: 20 tokens · 1.8×
GPT-2 tokenizer, reduced: precomputed numbers until it loads
Same question, five languages. Vietnamese costs almost four times the English tokens.
What a token is: tokens · token limits per minute: rate limits · what fills up: context window · rate-limit calculator
The same question costs 11 tokens in English and 40 in Vietnamese on GPT-2, because its pieces were learned from English; newer tokenizers shrink the gap to about 1.2×.
A tokenizer starts from bytes and learns, from its training text, which neighbours to glue into one piece: the most frequent pairs first. Text that was common ends up as whole words. Text that was rare, like “ợ” (3 bytes in UTF-8), stays in small pieces. Same meaning, more pieces, more tokens.
4 You are the model
Be the tokenizer. For each sentence, guess how many GPT-2 tokens it becomes, then see the pieces.
sentence 1 of 3 · how many GPT-2 tokens?
Hello, how are you?
GPT-2 tokenizer, reduced: precomputed numbers until it loads
Count words, then think: does every word get its own piece?5 Check
What to do about it: count your own language once (the gap is your multiplier). Prefer models with a newer tokenizer: GPT-4o’s o200k needs 12 tokens where GPT-4’s cl100k needs 22. Long fixed instructions can be in English, with “Answer in Vietnamese.”; the answer still costs Vietnamese tokens.
7 Say it back
8 Mission · 5 min
With a free Gemini API key, count the tokens of the English and the Vietnamese question from this page (count_tokens, Gemini docs “Understand and count tokens”). Then count the same question in your own language. Is Gemini’s gap closer to GPT-2’s 3.4× or to o200k’s 1.2× (the table above)?
Three quick ones
Drawn at random from this lesson's pool; different next time.
Done?
Go deeper
Sources
- GPT-2 counts in the picture and the predict step: the page’s reduced GPT-2 tokenizer (first 16,000 of 50,000 merges, app/lib/explain/gpt2-bpe.ts, #235), computed live; fallback numbers precomputed with the same code and checked by tests/explain-tokens-languages.test.ts
- Other tokenizers, measured 2026-10-22: tiktoken 0.14.0 encodings gpt2, cl100k_base, o200k_base; Hugging Face tokenizers 0.23.3 with https://huggingface.co/Qwen/Qwen2.5-7B-Instruct tokenizer.json (no special tokens). en/vi/es/pt-BR/id question: gpt2 10/34/16/14/18, cl100k 10/22/11/10/11, o200k 10/12/10/10/10, Qwen2.5 10/12/11/10/11; “Hello, how are you?” o200k 6, “Xin chào, bạn khỏe không?” o200k 8, “Resume este artículo en tres frases.” o200k 7
- https://github.com/openai/tiktoken/blob/main/tiktoken/model.py (checked 2026-10-22): gpt-4o, gpt-4.1, gpt-5, o1, o3 → o200k_base; gpt-4, gpt-3.5-turbo → cl100k_base
- Radford et al. 2019, “Language Models are Unsupervised Multitask Learners”, §3.7: “we deliberately removed non-English webpages from WebText as a filtering step”; a language detector found only 10MB of French
- UTF-8 byte lengths (TextEncoder, computed in the page): a 1, á 2, ư 2, ỗ 3
- Gemini count_tokens: https://ai.google.dev/gemini-api/docs/tokens; the Gemini and Llama 3 tokenizers were not measured here (account or API key needed), so no column for them
- The 1,000,000-token daily budget in the picture and pool is an example, labelled so
Published Oct 22, 2026 · updated Oct 22, 2026