Re-ranking, and When I Deliberately Skipped It: Retrieval Lessons From Building DocQA

Every RAG tutorial ends up in the same place: embed the question, embed the chunks, take the nearest few, stuff them in the prompt. Then, sooner or later, someone tells you the second stage — a cross-encoder re-ranker that re-orders those chunks by true relevance before the model reads them — is the cheapest big win you are leaving on the table. That advice is correct. But when I built DocQA, a small Claude-powered document Q&A app, I made a deliberate choice to skip both halves of that pipeline: no embeddings, no vector database, and no cross-encoder either.

This post is the honest version of the re-ranking story. I will explain what re-ranking actually buys you and why it works, and then I will show what a real retrieval layer looks like when you decide the vector-search-then-rerank machinery is more than your problem needs. DocQA ranks passages with a hand-written BM25 scorer and switches retrieval strategy based on document size — and being upfront about where it stops is the whole point.

What re-ranking actually fixes

The standard argument goes like this. Vector search uses a bi-encoder: it embeds the query and every document separately, ahead of time, and compares them by distance. That separation is exactly what makes it fast and scalable — and exactly what makes it coarse. The query and the document never look at each other; you are comparing two summaries made in isolation. A bi-encoder is good at recall (it reliably drags the right chunk into the top 50) and mediocre at precision (it often will not put that chunk in the top 5).

A cross-encoder fixes precision by reading the query and one candidate together and emitting a single relevance score, so it can judge subtle fit. The trade is that it cannot be precomputed and is far slower, so you only run it on the shortlist. Retrieve broadly and cheaply, then re-rank narrowly and expensively over a few dozen candidates. Same chunks, better order — the model reads the best evidence first instead of burying it at position nine.

Embeddings decide which documents are in the room; the re-ranker decides which ones get to speak. But before you reach for either, it is worth asking whether your corpus is small enough that the whole room fits in the prompt.

That last question is the one DocQA is built around. Re-ranking earns its keep when your corpus is large enough that you must retrieve a small slice, and when the first-stage retriever is noisy enough that the ordering inside that slice is wrong. Both conditions have to hold. For a single uploaded PDF or Markdown file, neither reliably does — and that changes the design.

DocQA ranks without embeddings, on purpose

DocQA answers questions about one document at a time: drop in a PDF, TXT, or Markdown file, ask a question, get an answer grounded in the text with clickable inline citations. The retrieval layer has two modes, and which one runs is decided by a single number — the document's character count.

• Small documents (under FULL_MODE_CHAR_LIMIT = 120,000 characters) skip retrieval entirely. The whole document goes into the prompt inside a cached system block, so the model sees everything and there is nothing to rank or miss.
• Larger documents switch to per-question retrieval: score every chunk against the question and keep the TOP_K = 8 best passages. No re-ranker runs after that — the ranker and the shortlist are the same step.

The ranker is a hand-written Okapi BM25 with its own tokenizer and stopword list, zero dependencies, no vector DB, no embeddings. BM25 is a lexical scorer: it rewards a chunk for containing the query's terms, weighted by how rare each term is across the document and dampened by chunk length. Here is the core of it, the k1 = 1.5, b = 0.75 loop straight out of the retriever:

const k1 = 1.5
const b = 0.75
for (const term of q) {
  const f = tf.get(term)
  if (!f) continue
  const n = df.get(term) || 0
  const idf = Math.log(1 + (N - n + 0.5) / (n + 0.5))
  const denom = f + k1 * (1 - b + (b * toks.length) / Math.max(avgdl, 1))
  score += idf * ((f * (k1 + 1)) / denom)
}

There is one small robustness detail worth calling out: if no chunk matches any query term, the retriever falls back to the first k chunks rather than returning nothing, so the model always has something to work with. That is the un-glamorous kind of edge case that only shows up once you point a retriever at real, messy input.

So where is the cross-encoder? There isn't one, and that is the honest answer. A cross-encoder re-ranker is a precision fix for a noisy first stage over a big corpus. DocQA's first stage is BM25 over the chunks of a single document, and its cheaper move is often to not retrieve at all. The 120k-character line is a cost decision as much as a size one: below it, the full document sits in a system block marked for prompt caching, so repeat questions about the same file read the cache. Anthropic's prompt-cache pricing puts cached reads at roughly ten percent of normal input cost — that is a documented pricing ratio the app leans on, not a benchmark I measured. Above the line, BM25's top-8 keeps token cost bounded at the price of the model only seeing eight passages.

const system =
  mode === 'full' && context
    ? [
        { type: 'text', text: SYSTEM },
        {
          type: 'text',
          text: 'DOCUMENT EXCERPTS:\n\n' + context,
          cache_control: { type: 'ephemeral' },
        },
      ]
    : SYSTEM

The retrieval quality you cannot skip: getting clean text

Here is the part the re-ranking discourse never mentions, because it assumes clean chunks fall from the sky. They do not. Long before ordering matters, you have to turn a PDF into text that is worth ranking at all — and pdf. js does not hand you clean text. It emits a stream of tiny positioned fragments, and a naive items. join(' ') produces garbage for anything that is not simple left-to-right Latin.

DocQA reconstructs text from geometry instead. For each fragment it looks at the x/y transform, width, and font size to decide whether the gap to the next fragment is a real word break, a line break, or nothing at all. The nasty case is complex scripts. For 'tight' scripts — Telugu and other Indic ranges, Thai, Arabic, CJK, Hangul — glyph clusters pack together with no inter-cluster space, and genuine word breaks arrive as literal space characters. If you use the Latin gap threshold there, you shatter words mid-cluster. So the threshold is raised from 0.25x font size to 0.9x for those ranges:

const tight =
  isTightScript(edgeCp(prev.str, true)) || isTightScript(edgeCp(it.str, false))
const spaceThreshold = tight ? 0.9 : 0.25

if (dy > fontSize * 0.5) {
  out += '\n' // new line even though hasEOL wasn't set
} else if (gap > fontSize * spaceThreshold) {
  out += ' ' // genuine inter-word gap
}
// otherwise the fragments are contiguous — concatenate with no space

This is a documented design rationale, not a benchmarked result — the repo has no test corpus proving extraction quality, and the Telugu case is an illustrative comment consistent with other work I have done on Indic text. There is a companion problem too: subsetted PDF fonts remap glyphs into the Unicode Private Use Area, which renders as black 'tofu' boxes. A codepoint-based sanitizer drops the Private Use Area (U+E000–F8FF), C0/C1 control characters, block-element boxes, and U+FFFD, while letting real scripts through untouched. The point is blunt: if your extractor mangles the text, no re-ranker downstream can save you. Ranking the wrong tokens perfectly is still wrong.

Grounding: the part re-ranking is really trying to protect

Re-ranking exists to get the best evidence in front of the model. But better evidence only helps if the model is actually forced to use it and cite it. DocQA spends its effort there instead. The system prompt forbids outside knowledge in plain language — if the answer is not in the excerpts, say you could not find it, never guess — and the answer must cite its sources inline with [n] markers.

Then the rendering layer defends against the model's own mistakes. Markdown is HTML-escaped first, then a constrained subset (code, bold, italic, lists) is re-applied and [n] markers become clickable citation chips that scroll to the exact source card that fed the answer — no untrusted HTML reaches the DOM. And because a model will occasionally emit a citation number higher than the number of sources it was actually given, any [n] where n exceeds the source count is clamped back to plain text, so it never becomes a dead or invented citation chip. Answers stream back as plain text/plain deltas — no SSE, no JSON envelope — and the browser never sees the Anthropic key, because an Express backend owns it and Vite proxies /api to it.

If you want to eval a RAG app, this contract is the thing to point evals at: answer only from the excerpts, cite [n], say I-could-not-find-it otherwise, and never let an out-of-range citation render. That is a concrete faithfulness target — and it matters whether your retriever is a cross-encoder stack or eight BM25 passages.

The honest bottom line

Re-ranking is real, and if you are running vector search over a large, noisy corpus, over-retrieve and drop a cross-encoder in front of the prompt — it is a few lines and one extra model, and it usually does more than swapping embedding models. But it is not a law of nature. DocQA is a working RAG app that never embeds anything: it caches the whole document when it is small, ranks with dependency-free BM25 when it is big, and spends the rest of its budget on getting clean text out of ugly PDFs and refusing to let the model cite a source that does not exist.

The lesson I would actually pass on is not 'add a re-ranker. ' It is: figure out which stage in your pipeline is lying to you. For a big corpus, it is often the ordering, and a re-ranker is the fix. For a single messy PDF, the lie is almost always upstream — in the text you extracted and the grounding you failed to enforce. Fix the stage that is wrong, not the stage the tutorials are excited about.