Chunking Strategies for RAG: The Decision That Quietly Makes or Breaks Retrieval

Everyone tuning a RAG app fiddles with prompts, embedding models, and re-rankers. Far fewer think hard about chunking — how you slice documents into pieces before embedding them — even though it silently caps how good retrieval can ever be. If the right answer is split across two chunks, or buried in a chunk full of unrelated text, no clever retrieval on top can fully recover it.

Chunking is the most boring decision in the pipeline and one of the most consequential. Get it right and every layer above it has clean, self-contained pieces to work with. Get it wrong and you spend weeks tuning around a problem you baked in on day one.

Too big and too small both fail

There is a real tension in chunk size, and both extremes hurt retrieval in different ways. The goal is a chunk that is small enough to be specific but large enough to stand on its own.

• Chunks too large dilute meaning. An embedding of a whole page averages many topics together, so it matches weakly to any one question and drags irrelevant text into the prompt.
• Chunks too small lose context. A single sentence often does not make sense alone — a pronoun with no referent, a number with no label — and embeds into something the query cannot find.
• Cutting mid-thought is the worst of both. Naive fixed-size splits slice through sentences and tables, leaving fragments that are neither specific nor complete.
• The sweet spot is usually a few hundred tokens of coherent, self-contained text — but it depends on your documents and embedding model, so it is something to measure, not guess.

Split on meaning, not character counts

The biggest upgrade over naive chunking is to split on the document's own structure instead of a fixed character count. Break on paragraphs, headings, list items, and code functions — the natural seams an author already put there — so each chunk is a coherent unit rather than an arbitrary window.

A recursive splitter does this well: it tries to break on the largest natural boundary first (say, paragraphs), and only falls back to smaller ones (lines, then sentences) when a piece is still too big. The result respects the writing instead of fighting it, and the chunks read like something a person would have separated.

A structural splitter, in code

In practice this is a few lines — set a target size, a little overlap, and let the splitter find the seams:

# Split on natural boundaries with a little overlap so nothing is lost at the seams.
from langchain_text_splitters import RecursiveCharacterTextSplitter

splitter = RecursiveCharacterTextSplitter(
    chunk_size=600,          # target size (characters or tokens)
    chunk_overlap=80,        # carry a little context across each boundary
)
# By default it recurses through paragraph, line, sentence, then word boundaries,
# so chunks break at natural seams instead of mid-sentence.
chunks = splitter.split_text(document)

# Attach metadata so a retrieved chunk still knows where it came from.
docs = [{"text": c, "source": path, "title": title} for c in chunks]

The overlap matters more than it looks: carrying a sentence or two across each boundary keeps an idea that straddles two chunks from being lost in the gap between them.

Retrieval can only find what chunking chose to keep together. Splitting documents is not preprocessing you rush through — it is the first and most permanent decision your RAG app makes.

The techniques that actually move the needle

• Overlap your chunks. A 10-to-20% overlap stops ideas that span a boundary from vanishing — cheap insurance against the worst failure mode.
• Respect structure. Split on headings, paragraphs, and code blocks; never cut through a table or a function. Format-aware splitters beat fixed windows on real documents.
• Carry metadata. Tag each chunk with its source, title, and section so you can filter, cite, and give the model context about where the text came from.
• Add context before embedding. Prepending a short summary of the document or section to each chunk — contextual retrieval — sharply improves matching for chunks that are ambiguous alone.
• Embed small, return big. Parent-document retrieval matches on small precise chunks but feeds the model the larger surrounding section, getting precision and context at once.

The bottom line

Chunking decides what your retriever is even able to find. Split on the document's natural structure, keep chunks coherent and a few hundred tokens, overlap the boundaries, and attach metadata — then evaluate retrieval on real queries and adjust. It is unglamorous work that pays off through every layer above it.

Before you swap embedding models or add a re-ranker, look at your chunks. If they are arbitrary slices of text, fix that first. Good retrieval starts with good pieces, and good pieces start with how you cut.