Chunking Strategy Decides Whether RAG Works

Most RAG systems fail quietly. Retrieval scores look fine in testing, but in production the answers drift, hallucinate, or miss context that's clearly in the document. The chunk is usually the culprit.

Chunking is not a pre-processing detail you settle once and forget. It is an architectural decision that determines what the retriever can and cannot see. Get it wrong and no amount of prompt engineering or model switching saves you.

Why Chunking Is the Real Retrieval Bottleneck

A vector index does not store documents. It stores chunk embeddings. When a user query comes in, the retriever finds the top-k most similar chunks and passes them to the LLM. If the chunk that contains the answer is split incorrectly, it either lands in two separate chunks with diluted signal, or gets merged with unrelated text that poisons the embedding.

The embedding model does not know what your document means. It knows what your chunk says. A 1,536-dimension vector from text-embedding-3-large is a compressed summary of the chunk it was built from. If the chunk is incoherent, the vector is too.

This is not a retrieval problem. It is a representation problem, and chunking controls it entirely.

What Are the Actual Chunking Strategies and When Do They Break?

The three strategies most teams reach for are fixed-size, recursive character splitting, and semantic chunking. Each has a failure mode that shows up in production.

Fixed-size chunking

You split every document into chunks of N tokens with an overlap of M tokens. LangChain's CharacterTextSplitter and TokenTextSplitter both do this. It is fast and deterministic.

The failure mode: it is structurally blind. A 512-token chunk from a legal contract can start mid-clause and end before the penalty condition. The embedding then describes an incomplete thought, and the retriever misses a query that would have matched the complete clause perfectly.

Overlap helps, but it is a patch. With 10–20% overlap you recover some context at the cost of storing redundant vectors that can confuse ranking.

Recursive character splitting

LangChain's RecursiveCharacterTextSplitter tries a hierarchy of separators: double newline, single newline, space, then character. It respects paragraph and sentence structure better than fixed-size splitting, and it is the default choice for a reason.

It still fails on structured documents. Tables, code blocks, numbered lists, and nested JSON get mangled because the separator hierarchy does not understand those formats.

Semantic chunking

You embed every sentence, compute cosine similarity between adjacent sentences, and split where similarity drops below a threshold. LlamaIndex ships a SemanticSplitterNodeParser that does exactly this. The chunks end up topic-coherent rather than structurally arbitrary.

The failure mode: it is slow (one embedding call per sentence), and it can create very uneven chunk sizes. A dense technical paragraph might stay whole at 600 tokens while a thin transitional paragraph becomes a chunk of 40 tokens. Both extremes hurt retrieval.

A practical comparison

Strategy	Speed	Structure awareness	Semantic coherence	Typical failure
Fixed-size	Fast	None	Low	Splits mid-thought
Recursive character	Fast	Partial	Medium	Breaks tables, code
Semantic	Slow	None	High	Uneven size, high cost
Document-aware	Medium	High	Medium-High	Complex to implement

Document-aware chunking, where you parse the document structure first (headings, sections, tables) and then chunk within those boundaries, is what production systems usually converge on. It requires format-specific parsers (Unstructured.io, Azure Document Intelligence, or custom PDF parsers using pdfplumber) but the retrieval quality improvement is significant.

Does Chunk Size Actually Matter That Much?

Yes, and the optimal size depends on your query type, not your document type.

Short factual queries ("What is the termination notice period?") retrieve better from small, precise chunks (128–256 tokens). The answer fits in one chunk and the embedding is specific enough to rank it high.

Long analytical queries ("Summarise the payment terms and explain how they interact with the liability cap") need larger chunks (512–1024 tokens) or a parent-document retrieval pattern. With small chunks, the retriever returns five fragments that each carry partial information, and the LLM either hallucinates the synthesis or hedges.

The parent-document retriever pattern from LangChain is useful here. You index small chunks for retrieval but return the parent chunk (larger window) to the LLM. You get retrieval precision and generation context at the same time. The trade-off is that you store documents at two granularities and your retrieval pipeline becomes stateful.

A chunk size of 512 tokens with 10% overlap is a reasonable starting point for mixed query types. It is not universally correct. Run evals before you commit.

Metadata Is Part of the Chunk

A chunk embedding does not encode where in the document the chunk came from. Without metadata, the retriever has no way to filter by section, date, document type, or source.

Every chunk should carry at minimum:

• Source document identifier
• Page or section number
• Document-level metadata (date, author, type) inherited at index time
• Chunk index within the document

Weaviate, Qdrant, and Pinecone all support metadata filtering. If you are not filtering on metadata before or after vector search, you are doing full-collection retrieval and paying for it in precision. A hybrid approach, where metadata filters narrow the candidate set and vector similarity ranks within it, consistently outperforms pure vector search on domain-specific corpora.

How Do You Know If Your Chunking Strategy Is Working?

You need an evaluation framework, not intuition.

The standard approach is to generate a question-answer dataset from your documents (GPT-4o can do this at scale), then run retrieval and score with RAGAS metrics: context precision, context recall, answer faithfulness, and answer relevance. RAGAS is open-source and integrates with LangChain and LlamaIndex.

Context recall is the metric most sensitive to chunking quality. It measures whether the retrieved chunks actually contain the information needed to answer the question. A score below 0.7 almost always points to a chunking or indexing problem, not a generation problem.

Run evals across chunk sizes (128, 256, 512, 1024 tokens) and strategies before you finalise your pipeline. The difference between a poorly chunked index and a well-chunked one can be 20–30 percentage points on context recall. That gap does not close by switching LLMs.

Conclusion

Chunking strategy sets the ceiling for everything downstream. A better retriever, a larger context window, or a more expensive embedding model will not compensate for a broken chunking approach. Start with document-aware splitting if your source documents have structure. Use semantic chunking if they do not. Run RAGAS evals with a representative question set before you ship. Adjust chunk size based on your query distribution, not a default from a tutorial.

If you are building a RAG system and retrieval quality is not where you need it, audit the chunks first. Check what the index actually contains. The answer is usually there.

---

FAQ

Does a larger context window make chunking less important? Longer context windows (128k tokens in GPT-4o, 200k in Claude 3.5) reduce some pressure on chunking by fitting more retrieved content. But retrieval precision still matters because cost and latency scale with tokens sent to the LLM. Poor chunking means more irrelevant tokens in the prompt and lower faithfulness scores, regardless of window size.

Can I re-chunk without rebuilding the entire index? Not cleanly. The chunk boundaries determine the embeddings, and the embeddings are the index. If you change your chunking strategy, you need to re-embed and re-index. Most teams maintain a document store separate from the vector index so re-indexing is a pipeline run rather than a manual process.

What chunk size works best for code documentation? Code and its surrounding documentation should stay together. Function signatures, docstrings, and inline comments belong in a single chunk. A function-level split is usually more useful than a token-count split. AST-based parsers like tree-sitter can provide clean boundaries for most languages.

Is semantic chunking worth the extra cost in production? It depends on document type. For well-structured documents (contracts, reports with clear headings), document-aware chunking gives similar quality at lower cost. For unstructured prose (research papers, support transcripts), semantic chunking's coherence gains justify the extra embedding calls. Benchmark both on your actual documents before deciding.

How does chunking interact with multi-modal RAG? For documents with images, tables, or charts, text-only chunking loses information. Tools like Azure Document Intelligence or Unstructured.io can extract table structure as markdown or JSON before chunking, which preserves relationships that plain text extraction flattens. Images need a separate pipeline: caption with a vision model, embed the caption, store the image reference.

Have a project in mind? Contact Sodio Technologies to discuss your requirements and explore the right technology solution for your business.