Chunking for RAG, Learned the Hard Way: What I Actually Chose Building DocQA

Everyone tuning a RAG app fiddles with the glamorous parts first: the prompt, the embedding model, the re-ranker. Far fewer people stop to think about chunking - how you slice a document into pieces before anything else touches it - even though that single decision quietly caps how good retrieval can ever be. If the answer to a question is split across two chunks, or buried in a chunk full of unrelated text, no amount of clever retrieval on top fully recovers it.

I ran into this head-on while building DocQA, a small Claude-powered document Q&A app: you drop in a PDF, TXT, or Markdown file, ask questions, and get answers grounded in the document with clickable inline citations that jump back to the exact source passage. It has no vector database and no embeddings - just a hand-rolled retriever and a chunker I had to tune by hand. That constraint made the chunking decisions unusually visible, so this post is the general theory plus exactly what I ended up choosing and why.

Too big and too small both fail

There is a real tension in chunk size, and both extremes hurt retrieval in different ways. The goal is a chunk small enough to be specific but large enough to stand on its own.

• Chunks too large dilute meaning. An embedding (or a keyword score) over a whole page averages many topics together, so it matches weakly to any one question and drags irrelevant text into the prompt.
• Chunks too small lose context. A lone sentence often makes no sense alone - a pronoun with no referent, a number with no label - and no query can find it.
• Cutting mid-thought is the worst of both. Naive fixed-size splits slice through sentences and tables, leaving fragments that are neither specific nor complete.

In DocQA I landed on a target of about 1600 characters - roughly 400 tokens - with a 200-character overlap between neighbours. I want to be honest about what that number is: it is a configuration constant I chose as a sensible default, not a value I benchmarked against a labelled retrieval set. There are no accuracy figures in the repo. It sits in the 'few hundred tokens of coherent text' range that tends to work, but on your own documents and embedding model it is something to measure, not inherit from me.

Split on meaning, not character counts

The biggest upgrade over naive chunking is to split on the document's own structure instead of a fixed character count. Break on paragraphs, headings, sentences, and code functions - the natural seams the author already put there - so each chunk is a coherent unit rather than an arbitrary window.

My chunker walks the text in windows of the target size, but instead of cutting hard at 1600 characters it looks backward for a paragraph or sentence boundary and cuts there. The one rule that saved me from silly output: only accept a boundary once you are past the halfway point of the chunk. Without that floor, a heading or a period near the very start of the window tempts the splitter into emitting tiny useless chunks; with it, every chunk is at least half the target size and still ends on a clean seam.

// DocQA chunker: ~1600-char target, 200-char overlap,
// only accept a boundary once we're past 50% of the chunk.
const TARGET = 1600   // ~400 tokens
const OVERLAP = 200

function pushChunk(text, page) {
  let start = 0
  while (start < text.length) {
    let end = Math.min(start + TARGET, text.length)
    // look backward for a paragraph or sentence seam,
    // but never before the halfway mark of this chunk.
    const floor = start + Math.floor((end - start) / 2)
    const seam = lastBoundary(text, floor, end)
    if (seam > floor) end = seam
    chunks.push({ text: text.slice(start, end), page })
    if (end >= text.length) break
    start = end - OVERLAP   // carry context across the cut
  }
}

The overlap matters more than it looks. Carrying 200 characters across each boundary keeps an idea that straddles two chunks from falling into the gap between them - cheap insurance against the single worst failure mode. I also thread the page number through every chunk, which is what lets a citation in the final answer say which page it came from and scroll the reader straight to it.

Retrieval can only find what chunking chose to keep together. Splitting a document is not preprocessing you rush through - it is the first and most permanent decision your RAG app makes.

The prerequisite nobody warns you about: getting clean text out of the PDF

Here is the gotcha that ate the most of my time, and it happens before chunking even starts. pdf. js does not hand you clean text. It emits a stream of tiny positioned fragments, and if you do the obvious thing - join them with spaces - you get spaces jammed into the middle of words and words fused together where there should be a break. Chunk that, and every chunk downstream inherits the damage.

So DocQA reconstructs text from geometry rather than joining strings. For each pair of fragments it looks at the horizontal gap and the vertical delta relative to the font size, and decides whether to insert nothing, a space, or a newline. The threshold that decides 'is this gap a real word break' is not one-size-fits-all. Latin text breaks words with gaps around a quarter of the font size; but 'tight' scripts - Telugu and other Indic ranges, Thai, Arabic, CJK, Hangul - pack glyph clusters with no inter-cluster space, and their real word breaks arrive as literal space characters. For those I raise the threshold most of the way to a full font-size before I'll call a gap a word break, or a naive join shatters the words.

const tight =
  isTightScript(edgeCp(prev.str, true)) || isTightScript(edgeCp(it.str, false))
const spaceThreshold = tight ? 0.9 : 0.25

if (dy > fontSize * 0.5) {
  out += '\n' // new line even though hasEOL wasn't set
} else if (gap > fontSize * spaceThreshold) {
  out += ' ' // genuine inter-word gap
}
// otherwise the fragments are contiguous - concatenate with no space

That Telugu rationale is a design comment in the code, not a benchmarked result - I have not measured extraction accuracy against a test corpus. But it is a real decision the extractor makes on every file. There was a second tar-pit too: subsetted PDF fonts map their glyphs into the Unicode Private Use Area, which renders as black 'tofu' boxes. The fix was a codepoint filter that drops the Private Use Area, control characters, block-element boxes, and the replacement character while leaving genuine scripts untouched.

The techniques that actually move the needle

Stepping back from DocQA's specifics, these are the levers that reliably help, roughly in the order I'd reach for them:

1. Overlap your chunks. A 10-to-20% overlap stops ideas that span a boundary from vanishing. My 200-character overlap on a 1600-character target is about 12%.
2. Respect structure. Split on headings, paragraphs, and sentences; never cut through a table or a function. In my chunker the 'only break past the halfway mark' rule is what keeps boundary-seeking from producing junk.
3. Carry metadata. Tag each chunk with its source and section - I carry the page number - so you can filter, cite, and tell the reader exactly where an answer came from.
4. Fix extraction first. On real PDFs, clean text out of pdf. js is a bigger win than any splitter tweak. Broken characters poison everything above them.
5. Add context before embedding. Prepending a short document or section summary to each chunk sharply improves matching for chunks that are ambiguous alone - a technique I have not added to DocQA yet, but would reach for next.

One design choice worth calling out, because it interacts with chunking: DocQA does not always retrieve. Below a 120,000-character cutoff it skips retrieval entirely and puts the whole document into a prompt-cached system block, so the model sees everything and repeat questions read the cache at roughly a tenth of the input cost. That cost ratio is Anthropic's documented prompt-cache pricing, not a benchmark I ran. Only above that line does it fall back to keyword retrieval over the chunks - which is exactly when good chunking starts to matter, because now the model only ever sees the top handful of passages the retriever picked.

The bottom line

Chunking decides what your retriever is even able to find. Split on the document's natural structure, keep chunks coherent and a few hundred tokens, overlap the boundaries, carry a little metadata - and, if you are dealing with PDFs, get clean text out of them before you split anything. Then evaluate on real queries and adjust the numbers, because mine are defaults I picked, not laws I measured.

Before you swap embedding models or bolt on a re-ranker, look at your chunks. If they are arbitrary slices of text - or worse, text with spaces in the middle of words - fix that first. Good retrieval starts with good pieces, and good pieces start with how you cut.