Why is DeepSeek so cheap (and why is a 671B model fast)?
Mixture of Experts: active vs total parameters · Beginner · 5 min · llm, transformer · after Model size, free vs paid tiers, Knowledge distillation (teacher and student models), Quantization (bits per weight), Model memory: parameters × bytes, plus the KV cache
A Mixture of Experts model splits its layers into many experts, and a router sends each token to only a few. DeepSeek-V3 stores 671B parameters but uses 37B per token, so it runs like a far smaller model.
1 Guess first
2 Look
Many experts, few at a time
1 / 5Mixtral layout: 8 experts per layer, the router picks 2 per token
token “def” → router → experts 2 and 5
E1E2E3E4E5E6E7E86 of 8 experts do nothing for this token. Next token, the router picks again.
Which experts light up here is illustrative; the real picks per token are in the Mixtral paper, figure 8.
Hugging Face configs and model cards, DeepSeek-V3 and Mixtral papers, Mistral, Groq · checked 2026-10-22
Tap a token. The router sends it to 2 of the 8 experts; the other 6 stay idle.
Why the total still needs memory: vram · free DeepSeek API · DeepSeek peak hours
A router sends each token to a few experts, so speed and price follow the parameters used per token, while memory still has to hold them all.
An expert is one feed-forward block inside a layer; an MoE layer has many side by side. The router is a tiny learned layer that scores them for each token and keeps the top few. Total parameters is everything stored; active is what one token passes through.
4 You are the model
You are the router. Send a line of code, a maths question and a line of poetry to the experts, then see what Mixtral’s router really did.
sentence 1 of 3 · Python code · you are the router
def total(self, items):
Mixtral of Experts paper (arXiv 2401.04088), §5 Routing analysis
Where would you send it? Then see what Mixtral’s router did.5 Check
Quality is not the active count either. Mixtral uses 13B per token yet matches or beats the dense Llama 2 70B on every benchmark its paper ran: the router can draw on all 47B. Cheap per token is not cheap to host, though: all 47B still sit in memory.
7 Say it back
8 Mission · 5 min
On Groq’s free tier, send the same prompt (“Explain recursion in 150 words”) to openai/gpt-oss-120b and openai/gpt-oss-20b. Divide usage.completion_tokens by usage.completion_time for each. The 120b is 5.6× bigger in total but only 1.4× in active parameters: which ratio is its slowdown closer to?
Three quick ones
Drawn at random from this lesson's pool; different next time.
Done?
Go deeper
Sources
- https://arxiv.org/abs/2412.19437 (checked 2026-10-22): “DeepSeek-V3 comprises 671B total parameters, of which 37B are activated for each token”; 61 layers, all FFNs except the first three replaced by MoE; “Each MoE layer consists of 1 shared expert and 256 routed experts”, “8 experts will be activated for each token”; full training 2.788M H800 GPU hours
- https://huggingface.co/deepseek-ai/DeepSeek-V3 config.json and safetensors totals (checked 2026-10-22): n_routed_experts 256, n_shared_experts 1, num_experts_per_tok 8, first_k_dense_replace 3; 680,571,043,840 F8_E4M3 + 3,918,786,560 BF16 parameters (684.5B incl. 14B multi-token prediction module) → 680.6 GB + 7.8 GB ≈ 688 GB of weights (arithmetic)
- https://mistral.ai/news/mixtral-of-experts (checked 2026-10-22): “Mixtral has 46.7B total parameters but only uses 12.9B parameters per token. It, therefore, processes input and generates output at the same speed and for the same cost as a 12.9B model.” Per expert (46.7 − 12.9) ÷ 6 ≈ 5.6B and shared ≈ 1.6B are arithmetic from these two numbers
- https://arxiv.org/abs/2401.04088 (checked 2026-10-22): 8 experts per layer, router selects two per token; “each token has access to 47B parameters, but only uses 13B active parameters”; “outperforms or matches Llama 2 70B and GPT-3.5 across all evaluated benchmarks”; §5: “we do not observe obvious patterns in the assignment of experts based on the topic”, ArXiv, PubMed and PhilPapers distributions “very similar”, DM Mathematics “marginally different”, “‘self’ in Python and ‘Question’ in English often get routed through the same expert”, indentation tokens “always assigned to the same experts”, consecutive tokens often share experts at layers 15 and 31
- https://huggingface.co/Qwen/Qwen3-235B-A22B (checked 2026-10-22): 235B in total and 22B activated; 94 layers; 128 experts, 8 activated
- https://huggingface.co/openai/gpt-oss-120b and /gpt-oss-20b, model cards and config.json (checked 2026-10-22): 117B parameters with 5.1B active, 128 experts, 4 per token, runs on a single 80GB GPU; 21B with 3.6B active, 32 experts, 4 per token, runs within 16GB of memory
- https://console.groq.com/docs/models (checked 2026-10-22): speed llama-3.3-70b-versatile 280 T/sec, openai/gpt-oss-120b 500, openai/gpt-oss-20b 1000
- aifree.dev catalog, groq-free-tier (verified 2026-10-12): Free plan includes openai/gpt-oss-120b and openai/gpt-oss-20b, 30 RPM, 1K RPD, 8K TPM
- The per-token expert picks in the first picture are illustrative, labelled as such; ratios 117 ÷ 21 ≈ 5.6 and 5.1 ÷ 3.6 ≈ 1.4, and 671 ÷ 37 ≈ 18, are arithmetic
Published Oct 22, 2026 · updated Oct 22, 2026