Before Multimodal RAG: What I Learned Getting Clean Text Out of Real PDFs

Every write-up on multimodal RAG jumps straight to the exciting part: a CLIP-style embedder that maps images and text into one vector space, or a vision-language model that reads a chart directly instead of choking on a mangled OCR transcript. That part is real and it works. But when I built DocQA, a small Claude-powered document Q&A app, I never reached the glamorous multimodal layer, because I got stuck on the boring thing nobody blogs about: getting clean text out of a real PDF is much harder than the tutorials admit, and if you get it wrong, every downstream stage inherits the garbage.

So this post is two things at once: the honest map of what multimodal RAG is and when you actually need it, and a first-hand tour of the unglamorous prerequisite I had to solve before any of that mattered - reconstructing readable text from the pile of tiny fragments pdf. js hands you, including complex scripts like Telugu and Arabic that shatter into broken words if you do the obvious thing. DocQA is deliberately text-only and ships no OCR or vision, which turns out to be the most useful part of the story: it shows exactly where the text pipeline stops and where the multimodal one has to begin.

Where text-only RAG actually breaks

The standard framing is right: most RAG systems assume your knowledge is clean text, and real documents are not. PDFs are full of charts, scanned forms, screenshots, and tables where the layout carries the meaning. The failure is upstream, at ingestion, because a text pipeline has to convert everything to characters first, and that conversion is where visual meaning dies.

• Charts and diagrams become nothing - a trend line has no text to extract, so it vanishes from the index.
• Tables lose their structure - flattening a grid into one line scrambles which number belongs to which row and column.
• Scans and screenshots depend on OCR, and imperfect OCR drops or garbles text while throwing away layout and emphasis.
• Sometimes the answer is the picture itself - for a receipt or a slide, describing the layout in text is a lossy copy.

Multimodal RAG is the fix for all of that: treat images as first-class content, retrieve the relevant pages or figures, and hand them to a model that can actually look. But notice a quiet assumption hiding in that list. Two of the four failures - charts and tables - are genuinely visual and need vision. The other two - scans and OCR quality - are really about whether you can recover the characters at all. And there is a fifth failure mode the tutorials skip entirely: documents that ARE text-based, that you can extract, and that STILL come out broken because the extractor was written for English and your document is not in English.

The prerequisite nobody blogs about: pdf. js does not give you clean text

Here is what actually happened when I started DocQA. The plan was mundane: extract text in the browser with pdf. js, chunk it, retrieve, answer. I expected extraction to be a solved, one-line problem. It is not. pdf. js does not hand you paragraphs - it emits a stream of many tiny text fragments, each with its own position and font, and it is entirely up to you to decide where a space goes, where a newline goes, and where two fragments are actually one word with nothing between them. The naive fix everyone reaches for is to join the fragments with a space. That single space is the bug.

For English you mostly get away with it. For a lot of the world you do not. Complex scripts - Telugu and other Indic scripts, Thai, Lao, Arabic, CJK, Hangul - pack glyph clusters together with no inter-cluster space, and their real word breaks arrive as literal space characters inside the fragment stream. Blindly joining every fragment with a space inserts spaces mid-word and shatters the text into nonsense. Your BM25 index then tokenizes that nonsense, your retriever matches the wrong things, and your model faithfully answers from garbage. The whole pipeline was poisoned at character one.

The fix I landed on was to stop treating extraction as string concatenation and start treating it as geometry. Instead of items. join, I look at each fragment's x/y transform, its width, and its font size, and I compute the gap to the previous fragment. A big vertical delta means a new line. A horizontal gap wider than a threshold means a genuine inter-word space. Anything smaller means the fragments are contiguous and get concatenated with nothing between them. The crucial twist is that the threshold is not constant: for tight scripts I raise the word-break gap from 0.25x the font size to 0.9x, precisely because those scripts have no small inter-glyph gaps and only the real, larger gaps should become spaces.

const tight =
  isTightScript(edgeCp(prev.str, true)) || isTightScript(edgeCp(it.str, false))
const spaceThreshold = tight ? 0.9 : 0.25

if (dy > fontSize * 0.5) {
  out += '\n' // new line even though hasEOL wasn't set
} else if (gap > fontSize * spaceThreshold) {
  out += ' ' // genuine inter-word gap
}
// otherwise the fragments are contiguous - concatenate with no space

I want to be careful about what I am claiming here. This is a documented design decision that lives in the extractor, and it is consistent with separate Telugu work I have done, but DocQA has no test corpus that benchmarks extraction quality. So read this as a real design rationale I implemented, not as a measured accuracy number - there is no such number in the repo, and I am not going to invent one.

Before you reach for a vision model, make sure you are not shattering the text you already have. Half of what looks like a multimodal problem is really a broken extractor written for English and handed a document that was not.

The other extraction gremlin: tofu boxes from subsetted fonts

The second thing that ambushed me was tofu - those little black squares that show up where readable characters should be. The cause is that PDFs frequently embed subsetted fonts that remap their glyphs into the Unicode Private Use Area. When you extract that text, you get valid-looking codepoints in the U+E000 to U+F8FF range that render as boxes, plus stray control characters and replacement characters that pollute the same stream. If those leak into your chunks and your prompt, you are spending tokens on noise and confusing the model.

My answer was a codepoint-based sanitizer that filters by numeric range rather than matching exotic literal characters in the source. It drops C0/C1 control characters, the Private Use Area (the usual cause of the black squares), block-element boxes in the U+2580 to U+259F range, and the U+FFFD replacement character - while deliberately leaving every genuine script untouched. The filter-by-number detail matters in practice: it keeps the source file clean and ASCII-friendly, and it means real Telugu, Arabic, or CJK text passes straight through because it never lands in one of those junk ranges.

None of this is exciting. All of it is load-bearing. Extraction is the stage where, if you cut a corner, nobody sees the corner - they just see a chatbot that gives subtly wrong answers and blame the model.

Where DocQA stops, and where multimodal has to start

Here is the honest boundary. Once the text was clean, the rest of DocQA is deliberately modest: a page-aware overlapping chunker (roughly 1600-character targets, about 400 tokens, with 200 characters of overlap and page numbers threaded through so a citation can name a page), and then one of two context strategies chosen purely by document size.

1. Small documents, under a 120,000-character limit, go into the prompt whole inside a cached system block, so repeat questions about the same doc read the cache at roughly 10% of the input cost. That ratio reflects Anthropic's prompt-cache pricing, not a benchmark I ran.
2. Larger documents switch to a hand-rolled, dependency-free Okapi BM25 retriever (k1 = 1.5, b = 0.75) that returns the top 8 passages per question - no vector database, no embeddings - with a fallback to the first k chunks when nothing matches so the model always has something to work with.

That is a real, complete RAG pipeline, and it is also unmistakably text-only. Scanned or image PDFs are explicitly unsupported - there is no OCR and no vision model in the box. This is exactly the line where multimodal RAG has to take over. If your document is a photo of a receipt, DocQA has nothing to extract, and no amount of clever geometry helps; you need OCR to recover characters or a vision-language model to read the pixels. My text pipeline is the correct tool right up to the moment the information stops being text, and then it is the wrong tool completely.

So if you are designing a multimodal RAG system, the practical takeaway from my build is to route, not to blanket. Send true images and scans to OCR or a vision model. Send charts and layout-heavy pages to a model that can see. But send the large, boring, text-based-but-non-English majority of real documents through a text path - and make that text path good enough to survive glyph geometry and subsetted fonts, because that is where most of the silent quality loss actually happens.

The bottom line

Multimodal RAG is genuinely the right answer when layout is the meaning: keep the visuals intact, retrieve the relevant pages, and let a model that can see do the reading. But before you buy the vision layer, audit the boring stage first. When I built DocQA I discovered that a large share of what people call a multimodal problem is really a text-extraction problem in disguise - words shattered by a naive fragment join, tofu boxes from subsetted fonts, structure lost because the extractor never looked at the geometry.

Fix the extractor and a surprising number of your documents become answerable without a single image embedding. Then, and only then, reach for multimodal RAG for the documents that genuinely are pictures. Build a pipeline that knows which kind of document it is holding, and route each one to the tool that can actually read it.