How does it turn my voice into text?
Speech-to-text: spectrograms and audio tokens (Whisper) · Beginner · 5 min · tokens, transformer · after Tokens, Vision: images as tokens
A speech-to-text model like Whisper turns sound into a picture (a spectrogram of pitch over time), cuts it into audio tokens, one every 20 ms, and a decoder then writes the text token by token, like a chatbot.
1 Guess first
2 Look
From your voice to text
1 / 41 s of your voice = 16,000 numbers
A microphone measures air pressure; Whisper re-samples every recording to 16,000 measurements a second. No words, no letters: just numbers going up and down.
Whisper: OpenAI’s whisper/audio.py and Radford et al. 2022 §2.2 · checked 2026-10-23
Sound arrives as numbers: 16,000 a second, nothing that looks like a word.
The model never hears words: sound becomes numbers, then a picture of pitch over time, then 1,500 audio tokens, and only then does a decoder write text.
This is the same trick as vision: turn the input into a grid, cut the grid into tokens, and let a transformer read them. The decoder is a language model that listens: it predicts the next text token from the audio tokens and the text so far. When the sound is unclear, it picks the likeliest words.
4 You are the model
You are Whisper’s input stage. For four recordings, decide how many audio tokens each one becomes.
you are Whisper’s input stage · 30-second windows · one audio token per 20 ms
3-second “yes, sounds good”: how many audio tokens?
Whisper: OpenAI’s whisper/audio.py and Radford et al. 2022 §2.2 · checked 2026-10-23
You are the input stage. How many audio tokens does this clip become?5 Check
Two habits help. Tell it the language: it is one token in the prompt, and guessing it can go wrong. And watch silence and window edges: the Whisper paper lists repeat loops, missing first or last words of a segment, and “complete hallucination” of text that was never said.
7 Say it back
8 Mission · 5 min
With Groq’s free plan, send a 10-second voice memo to whisper-large-v3-turbo, once with language set to your language and once set to a different one. Then send 15 seconds of silence. Does the wrong language token change the text, and does the silence come back empty or with words?
Three quick ones
Drawn at random from this lesson's pool; different next time.
Done?
Go deeper
Sources
- https://arxiv.org/abs/2212.04356 (Radford et al. 2022, checked 2026-10-23): §2.2 audio re-sampled to 16,000 Hz, 80-channel log-magnitude Mel spectrogram on 25-millisecond windows with a stride of 10 milliseconds; stem of two convolution layers, the second with a stride of two; GPT-2 byte-level BPE for the English-only models
- Same paper: §2.1 30-second segments; §2.3 <|startoftranscript|>, a language token (99 languages), <|transcribe|> or <|translate|>, <|notimestamps|>, timestamps quantized to 20 milliseconds; §3.8 trained on 30-second chunks, long audio transcribed by consecutive 30-second segments; abstract 680,000 hours; §6 limitations: repeat loops, not transcribing the first or last few words of a segment, “complete hallucination … entirely unrelated to the actual audio”
- https://github.com/openai/whisper/blob/main/whisper/audio.py (checked 2026-10-23): SAMPLE_RATE 16000, HOP_LENGTH 160, CHUNK_LENGTH 30, N_SAMPLES 480000, N_FRAMES 3000, N_SAMPLES_PER_TOKEN = HOP_LENGTH * 2 (“the initial convolutions has stride 2”), TOKENS_PER_SECOND 50 (“20ms per audio token”), pad_or_trim to N_SAMPLES
- https://huggingface.co/openai/whisper-large-v3 (checked 2026-10-23): “uses 128 Mel frequency bins instead of 80”; trained on 1 million hours of weakly labeled audio and 4 million hours of pseudo-labeled audio
- Window arithmetic (3 s, 30 s → 1,500; 45 s → 3,000; 130 s → 7,500 audio tokens) and the memo’s 72 text tokens (55 words, full GPT-2 BPE) from scripts/explain-data/speech-to-text.ts, run 2026-10-23; the memo was written for the lesson
- https://console.groq.com/docs/speech-to-text (checked 2026-10-23): whisper-large-v3 and whisper-large-v3-turbo, audio downsampled to 16 kHz mono, a 10-second minimum billed per request, `language` parameter in ISO-639-1; aifree.dev catalog groq-free-tier (verified 2026-10-12) and /learn/free-groq-api-key: Whisper on the Free plan at 20 RPM, 2,000 RPD, 7,200 audio seconds per hour
Published Oct 23, 2026 · updated Oct 23, 2026