Why does RAG return the wrong paragraph?
Chunking: chunk size, overlap and boundaries in RAG · Builder · 5 min · rag, embeddings · after Semantic search: embeddings and cosine similarity
RAG cuts documents into chunks and returns the chunk whose embedding points closest to the question. Where the cuts fall decides whether that chunk holds the answer: a cut mid-idea makes a chunk that sounds right but lacks it.
1 Guess first
2 Look
Where the cuts fall
1 / 4ask:“How many requests per day does the free plan allow?”
Every account starts on the Free plan.
The Pro plan costs $20 a month, billed in advance. It allows 50,000 requests per day, and support answers within four hours.
The Free plan needs no card: sign up with an email address and your API key appears at once. It allows 1,000 requests per day.
Requests over the daily limit return error 429 until midnight UTC, when the counter resets. Unused requests do not roll over.
Team plans start at five seats. Each seat adds 10,000 requests per day to a shared pool.
5 chunks · the search returns the lit one, score 0.75: the answer, “1,000”, is not in it ✗
- ✗ requests per day, and support answers within four hours. The Free plan needs no card: sign up with an email0.75
- ✓ address and your API key appears at once. It allows 1,000 requests per day. Requests over the daily limit return0.64
- ✗ Every account starts on the Free plan. The Pro plan costs $20 a month, billed in advance. It allows 50,0000.57
measured with all-MiniLM-L6-v2 (384 numbers per chunk) · cosine similarity · 2026-10-23
Cut every 20 words: the free-plan question gets a chunk with neither number. Drag the size.
The search returns the chunk that sounds most like the question; if a cut splits the idea, that chunk can sound right and hold nothing.
A RAG app cannot embed a whole manual as one vector, so it cuts documents into chunks, embeds each, and pastes the closest few into the prompt. The search scores how much a chunk *sounds like* the question, not whether it answers it. Where the cuts fall decides what a chunk is about.
4 You are the model
You are the search. For four questions, each on the page cut a different way, pick the chunk all-MiniLM-L6-v2 scores highest.
you are the search · pick the chunk all-MiniLM-L6-v2 scores highest
“When does the daily request limit reset?” · cut every 20 words
measured with all-MiniLM-L6-v2 (384 numbers per chunk) · cosine similarity · 2026-10-23
You are the search. Which chunk scores highest?5 Check
Habits that help. Cut along the document’s own structure: headings, paragraphs, sentences. Keep one idea per chunk, and a little overlap for ideas that run across a cut. Add the title or section to each chunk; Anthropic’s tests cut failed retrievals by 35% that way. Then test with real questions: does the top chunk hold the answer?
7 Say it back
8 Mission · 5 min
Copy the free key on Jina’s embeddings page (no card) and embed three texts in one request with "normalized": true: the question “How many requests per day does the free plan allow?”, the 20-word piece “requests per day, and support answers within four hours. The Free plan needs no card: sign up with an email”, and the paragraph “The Free plan needs no card: sign up with an email address and your API key appears at once. It allows 1,000 requests per day.” Dot each with the question. Does a bigger model make the same mistake?
Three quick ones
Drawn at random from this lesson's pool; different next time.
Done?
Go deeper
Sources
- Cosine similarities measured with sentence-transformers/all-MiniLM-L6-v2 (https://huggingface.co/sentence-transformers/all-MiniLM-L6-v2: 384 dimensions, mean pooling, normalised, max 256 word pieces) on 2026-10-23 by scripts/explain-data/chunking.ts, which reuses the plain forward pass of scripts/explain-data/semantic-search.ts (it reproduces the published sbert.net example 0.6660, 0.1046, 0.1411)
- The pricing page (92 words, 5 paragraphs, a made-up API) and the three questions were written for the lesson; the cuts are computed by app/components/explain/lessons/chunking-data.ts, the same code the picture uses
- Scores used: 20 words, free plan → mixed Pro/Free chunk 0.753, chunk with 1,000 0.637, first chunk 0.572; 20 words, reset → 1,000 chunk 0.689, midnight chunk 0.507; 20 words + 10 overlap, reset → 0.784 (holds the answer), free → 0.753 (mixed chunk again); 10 words, Pro price → “Every account … The Pro plan” 0.654, “costs $20 …” 0.511; by sentence, free → “It allows 1,000 requests per day.” 0.759, “It allows 50,000 …” 0.680; by paragraph → free 0.780, reset 0.784, Pro 0.836; whole page → free 0.764, reset 0.413, Pro 0.766
- https://www.anthropic.com/news/contextual-retrieval (Sep 19, 2024, checked 2026-10-23): Contextual Embeddings cut the top-20 retrieval failure rate by 35% (5.7% → 3.7%); with Contextual BM25 by 49%; “The choice of chunk size, chunk boundary, and chunk overlap can affect retrieval performance”
- https://jina.ai/embeddings/ (checked 2026-10-23): new users get an auto-generated API key with free tokens; endpoint https://api.jina.ai/v1/embeddings; aifree.dev catalog jina-ai-free-api-tokens (verified 2026-10-12): 10M tokens per new API key, no card
Published Oct 23, 2026 · updated Oct 23, 2026