Agentic RAG, and Why I Deliberately Did Not Build It Into My Document Q&A App

Everyone writes the same post about agentic RAG: hand retrieval to the model as a tool, let it rewrite the query, search again, and stop when it has enough. It is a genuinely good pattern. But I want to write the version I wish I had read before I started, because when I actually built a document Q&A app called DocQA, the most useful lesson was almost the opposite of the hype: for a huge class of questions, the smartest agentic loop I could design was to not have one at all.

DocQA is small on purpose. You drop in a PDF, TXT, or Markdown file, ask questions, and get answers grounded in that document with clickable inline citations that jump back to the exact passage that produced them. React and Vite on the front end, a thin Express server that owns the Anthropic key, Claude on the back end. No vector database, no embeddings, no framework. That constraint is what taught me where agentic retrieval actually earns its cost, and where it is pure overhead.

What agentic RAG really is

Classic RAG is a straight line. Take the question, run one retrieval, stuff the results into the prompt, generate an answer. It works beautifully when the answer lives in a few chunks that a single search can find, and it falls apart the moment a question needs evidence from several places or the first search simply misses.

Agentic RAG turns that line into a loop. Instead of retrieving for the model, you expose search as a tool and let the model drive: rephrase a vague question into a better query, search more than once, judge whether what came back is enough, route to a different index, or skip retrieval entirely for a greeting. The model owns the decision of what to look up and when to stop. That unlocks multi-hop questions and self-correction that one-shot retrieval cannot touch.

It also costs you. Every retrieval round is another model call, so a multi-hop answer is slower and pricier. Without a hard step cap the loop can spin in circles. And because the retrieval path now varies per question, you need tracing to understand what the model searched and why it stopped. The honest framing is that agentic RAG is an upgrade you reach for when single-shot retrieval keeps missing, not a default.

The interesting decision in a RAG app is not how clever your loop is. It is knowing which questions deserve a loop at all — and being willing to answer most of them without one.

The baseline I actually shipped

Before you let a model decide what to retrieve, you have to decide what it even gets to see. In DocQA that is a single branch on document size, and it is a cost decision as much as a size one.

• Under 120,000 characters (FULL_MODE_CHAR_LIMIT), the whole document goes into the prompt as a cached system block — 'full' mode. The model sees everything, so there is nothing to retrieve and nothing to get wrong.
• Above that line, I fall back to per-question retrieval: a hand-written Okapi BM25 pass picks the TOP_K = 8 best passages, and only those go into the prompt. Token cost stays bounded, at the price of the model seeing only eight passages instead of the whole thing.

The reason full mode is worth having is prompt caching. The document text is a second system block marked with cache_control ephemeral, so repeat questions about the same document read the cache instead of re-billing the whole thing. Here is the actual server code:

const system =
  mode === 'full' && context
    ? [
        { type: 'text', text: SYSTEM },
        {
          type: 'text',
          text: 'DOCUMENT EXCERPTS:\n\n' + context,
          cache_control: { type: 'ephemeral' },
        },
      ]
    : SYSTEM

Anthropic's pricing means a cached read comes in at roughly 10% of the input cost, and the README and my server logs lean on that. I want to be precise about what that number is: it is the published prompt-cache pricing ratio, not a benchmark I ran. DocQA has no latency numbers, no retrieval-accuracy figures, and no eval results — everything I am quoting here is a configuration constant or that one documented cost ratio. I did not measure answer quality; I read and built the code, I did not run a controlled evaluation of it.

Why I did NOT build the agentic loop

Every generic post ends with 'and then you make it agentic. ' DocQA deliberately stops before that, and sitting with the decision taught me more than adding the loop would have. For a single uploaded document, the two failure modes agentic RAG fixes mostly do not fire. Full mode already hands the model the entire document, so there is no first-try miss to recover from — the model is not searching at all. And a hand-tuned BM25 pass over one document is a genuinely strong baseline for the retrieval mode; the questions people ask a single PDF are usually answerable from a good top-8, not from three chained searches across separate indexes.

Where an agentic loop would clearly pay off is exactly where DocQA draws its line: above 120k characters, in retrieval mode, when a question needs evidence that BM25's top-8 splits across passages it ranked ninth and tenth. That is the honest place to add query rewriting and a second search. It is on my list. But adding it below that line — to full mode, where the model already sees everything — would be pure latency and token cost for zero recall gain. The loop is a tool with a bill attached, and most of DocQA's traffic does not need to pay it.

The retriever itself is deliberately boring: dependency-free Okapi BM25 with k1 = 1.5 and b = 0.75, its own tokenizer and stopword list, and a fallback to the first k chunks when no term matches so the model always has something to read. No vector DB, no embedding service, nothing to keep warm.

const k1 = 1.5
const b = 0.75
for (const term of q) {
  const f = tf.get(term)
  if (!f) continue
  const n = df.get(term) || 0
  const idf = Math.log(1 + (N - n + 0.5) / (n + 0.5))
  const denom = f + k1 * (1 - b + (b * toks.length) / Math.max(avgdl, 1))
  score += idf * ((f * (k1 + 1)) / denom)
}

The gotchas that actually ate my time

Here is the part no agentic-RAG explainer prepares you for. The clever retrieval loop is not where the hard problems were. They were one layer down, in getting clean text out of a real PDF and getting a citation to point at the right place. If your input text is broken, no amount of agentic reasoning saves you — garbage in, confidently-cited garbage out.

• pdf. js does not hand you clean text. It emits many tiny fragments, and a blind join with spaces inserts spaces mid-word. I rebuild text from each fragment's geometry — the x/y transform, width, and font size — to decide whether to insert a space, a newline, or nothing.
• Complex scripts shatter into pieces. For 'tight' scripts like Telugu, Arabic, and CJK, glyph clusters pack together with no inter-cluster space, so I raise the word-break gap threshold from 0.25x to 0.9x of font size. Miss this and words break apart mid-token.
• Subsetted fonts render as black 'tofu' boxes because they map glyphs into the Unicode Private Use Area. A codepoint filter drops the PUA (U+E000 to U+F8FF), control chars, block-element boxes, and U+FFFD while leaving real scripts untouched.
• The model can cite a source that does not exist. If it emits a citation number higher than the number of sources it was given, the Markdown renderer clamps that out-of-range marker back to plain text so it never becomes a dead, hallucinated chip.

That geometry-based join is the single most consequential piece of code in the app, and it has nothing to do with retrieval strategy:

const tight =
  isTightScript(edgeCp(prev.str, true)) || isTightScript(edgeCp(it.str, false))
const spaceThreshold = tight ? 0.9 : 0.25

if (dy > fontSize * 0.5) {
  out += '\n' // new line even though hasEOL wasn't set
} else if (gap > fontSize * spaceThreshold) {
  out += ' ' // genuine inter-word gap
}
// otherwise the fragments are contiguous — concatenate with no space

The tight-script handling is a documented design rationale, consistent with separate Telugu work I have done — not a benchmarked win. DocQA has no test corpus proving extraction quality. I am telling you the reasoning behind the code, not showing you a measured result, and that distinction matters if you are deciding whether to copy the approach.

What I would actually tell you

Agentic RAG is the right upgrade when correctness genuinely depends on gathering evidence across several searches: multi-hop questions, ambiguous phrasing, multiple sources, answers that need the model to look, think, and look again. Give search to the model as a tool, cap the loop, trace it, and you get retrieval that adapts to the question.

But do not reach for it by default, and do not let the pattern's elegance distract you from the boring layers underneath it. Building DocQA convinced me that the highest-leverage decisions in a small RAG app are upstream of the loop: whether to retrieve at all, how to keep repeat questions cheap with caching, and whether the text you extracted is even correct. Get a clean baseline that answers most questions well, ground it hard so the model cannot invent sources, and add the agentic loop only where you can point at the exact questions it would rescue. For me, that place exists — it is just a lot narrower than the internet makes it sound.