Key facts.
- Chunking strategy can swamp everything else: a clinical-RAG study found structure-aligned chunking hit 87% accuracy versus 13% for a fixed-size baseline on the same documents (MDPI Bioengineering, 2025).
- The gap shows up in recall too: fixed-size scored 0.40, semantic chunking scored 0.75, the fixed-size pipeline missed the relevant material far more often (same study, 2025).
- There's no universal right size: NVIDIA's 2024 evaluation across seven strategies and five datasets found the best size depends on query type, factoid questions around 256-512 tokens, analytical questions 1024 or more (NVIDIA, 2024).
What does bad chunking actually do?
It cuts facts in half. Documents get split into chunks before anything is retrieved. A fixed-size splitter cuts at arbitrary token boundaries, no regard for meaning. A rule lands in one chunk, its exception in the next. A table row gets split from its header. A definition gets split from the term. Retrieval returns one half, and the model answers from a partial truth, confidently. The source had the whole fact. The chunker broke it before the retriever ever saw it. That's why the agent gives a half-answer that's technically grounded in your data and still wrong.
# A fixed 256-token split lands mid-rule. Two chunks, neither complete: --- chunk 41 --- "Returns are accepted within 30 days of delivery, --- chunk 42 (boundary cut here) --- except for final-sale items, which are non-refundable." # Retrieve chunk 41 and the agent says "30-day returns", missing the exception.
Why one chunk size always loses
One chunk has two jobs that pull in opposite directions. Sharp retrieval wants small, focused chunks, 100-256 tokens, one clear topic, tight embedding. But the model needs context completeness, surrounding text, 1000+ tokens, so it doesn't miss a caveat. A fixed size can't satisfy both. Go small: retrieval works, the model gets fragments. Go large: chunks blur across topics, retrieval struggles. That's the structural conflict in naive chunk-embed-retrieve, and it's why no global chunk size is ever right.
How big is the gap?
Large enough to dominate everything downstream. Clinical decision-support evaluation: structure-aligned chunking hit 87% accuracy, fixed-size baseline hit 13% on the same documents. Recall matched: 0.75 vs. 0.40. NVIDIA tested seven strategies across five datasets, factoid queries did best at 256-512 tokens, analytical queries needed 1024+, page-level was a strong general default. No magic number. What the results say: chunking is a first-class design decision, and getting it wrong caps your accuracy no matter how good the model or retriever is.
How do you chunk so facts stay whole?
| Technique | What it does |
|---|---|
| Semantic / structure-aware chunking | Split at topic, section, and document boundaries, not token counts |
| Respect tables and lists | Keep a row with its header and an item with its context |
| Add overlap | Let chunks share a window so a fact split at a boundary survives in one of them |
| Attach metadata | Tag each chunk with its source, section, and date for filtering and grounding |
| Parent-document retrieval | Match on small chunks, then return the larger parent for context |
| Late chunking | Embed the long document first, then chunk, so each chunk keeps document context |
Fixed-size chunking severs the fact from context; a better model reads the same half. (arXiv:2406.00456)
The pattern is that retrieval can only return what chunking left whole, and a fixed-size splitter routinely cuts facts in half before the model ever sees them. Split at meaning, keep structure intact, overlap the boundaries, and retrieve small but read large with parent-document or late chunking. None of that's a bigger model, which only paraphrases the fragment more fluently. it's a data layer that keeps facts intact from source to context, which is what VibeModel builds as the Pattern Intelligence Layer.
Frequently asked questions
Isn't this fixed by a bigger context window?
No. The damage happens at retrieval: if the chunk that was fetched is a fragment, a bigger window just holds a bigger fragment. You have to chunk so the whole fact is retrievable in the first place, then use the window to read it in full context.
What chunk size should I use?
It depends on the query. Factoid lookups do well around 256 to 512 tokens; analytical questions need 1024 or more. Rather than pick one, retrieve on small chunks and return the larger parent, so you get sharp matching and full context.
What's the single highest-impact change?
Stop splitting at fixed token counts. Switch to structure-aware chunking that respects sections, tables, and topic boundaries, and add overlap. That alone closes most of the gap between a 13% and an 87% pipeline.