Why does the answer appear word by word?
Streaming, time to first token and tokens per second · Beginner · 5 min · tokens, llm · after Next-token prediction, Rate limits (RPM, TPM, RPD)
Streaming means an AI API sends each token the moment the model makes it. The model writes one token per step, so the wait has two parts: time to the first token, then tokens per second.
1 Guess first
2 Look
One token per step
1 / 5your question · 6 tokens
- Why
- ·is
- ·the
- ·sky
- ·blue
- ?
answer · 0 of 11 tokens made
The whole question is read in one pass. No answer yet.
tokens: GPT-2 tokenizer, reduced · question and answer text illustrative
Your question goes in whole: six tokens, read in one pass.
Busy hours are real: DeepSeek’s peak hours in your time zone
The model makes one token per step and each step reads everything before it; streaming shows each token as soon as it is made, after a wait in line and one pass over your question.
Two numbers describe the wait. Time to first token (how long until the first word) is the queue plus reading your prompt. Tokens per second (how fast steps follow) sets the rest. Groq lists 1,000 tokens per second for gpt-oss-20b and 280 for llama-3.3-70b: 500 tokens take 0.5 or about 1.8 seconds.
4 You are the model
You are the model. Make the answer one token per press, and watch two users: one with streaming on, one with it off.
your question · 6 tokens
- Why
- ·is
- ·the
- ·sky
- ·blue
- ?
answer · 0 of 11 tokens made
The whole question is read in one pass. No answer yet.
user’s screen · stream: true
… still waiting
user’s screen · stream: false
… still waiting
tokens: GPT-2 tokenizer, reduced · answer text illustrative
You are the model. Make the answer, one token per press.5 Check
A free tier shares its chips with many people. At a busy hour your request can wait in a queue (the line before step one), so the first word comes late, and steps can slow while more requests share the chips. Some vendors price busy hours higher: DeepSeek charges double on weekday peak hours.
7 Say it back
8 Mission · 5 min
With a free Groq key, send “Explain why the sky is blue.” to openai/gpt-oss-20b twice: once with "stream": true and watch the words arrive, once without. In the second response, read usage: queue_time, prompt_time and completion_time, then divide completion_tokens by completion_time.
Three quick ones
Drawn at random from this lesson's pool; different next time.
Done?
Go deeper
Sources
- https://huggingface.co/docs/transformers/main/en/llm_tutorial (checked 2026-10-22; no update date): an LLM generates “the next word (token) given some initial text (prompt) along with its own generated outputs”
- https://huggingface.co/docs/transformers/main/en/llm_optims (checked 2026-10-22; no update date): “LLMs are called repeatedly to generate the next token”
- https://console.groq.com/docs/api-reference (checked 2026-10-22; no update date): stream “Tokens will be sent as data-only server-sent events as they become available, with the stream terminated by a data: [DONE] message”; example response usage queue_time 0.037493756, prompt_tokens 18, prompt_time 0.000680594, completion_tokens 556, completion_time 0.463333333 (556 ÷ 0.463 ≈ 1,200 tokens per second)
- https://console.groq.com/docs/models (checked 2026-10-22; no update date): speed openai/gpt-oss-20b 1000 T/sec, llama-3.3-70b-versatile 280 T/sec
- https://api-docs.deepseek.com/quick_start/pricing (checked 2026-10-21, app/lib/peak-hours.ts): peak hours 01:00–04:00 and 06:00–10:00 UTC Monday to Friday, peak price 2× off-peak on every row
- aifree.dev catalog: groq-free-tier verified 2026-10-12, free plan with gpt-oss-20b (30 RPM, 8K TPM)
- The question and answer text and the busy-hour queue length in the picture are illustrative; the tokens come from the GPT-2 tokenizer (reduced, app/lib/explain/gpt2-bpe.ts)
Published Oct 22, 2026 · updated Oct 22, 2026