Agentic RAG: Letting the Model Decide What to Retrieve

Standard RAG is a straight line: take the user's question, run one retrieval, stuff the results into the prompt, generate an answer. It works beautifully when the answer lives in a few chunks that a single search can find. It falls apart the moment a question needs information from several places, or when the first search simply misses.

Agentic RAG turns that straight line into a loop. Instead of retrieving for the model, you give the model retrieval as a tool and let it drive: rephrase a vague question, search more than once, judge whether what came back is enough, and only answer when it is. It trades simplicity and speed for the ability to actually handle hard questions.

Where one-shot retrieval breaks

A single retrieval on the raw question assumes the question is already a good search query and that one search suffices. Real questions violate both assumptions constantly, and classic RAG has no way to recover when they do.

• Multi-hop questions. Comparing two things, or chaining facts, needs several retrievals — one search cannot gather them all.
• Bad query phrasing. The user's wording may not match the documents; the model could rewrite it into a far better search, but classic RAG never gets the chance.
• First-try misses. If the top chunks are wrong, plain RAG answers from them anyway, with no second look.
• Retrieval that is not even needed. For a greeting or a reasoning question, classic RAG retrieves regardless, adding noise and cost.

Retrieval as a tool the model controls

Agentic RAG is built on tool calling. Search becomes a function the model can invoke with a query it writes itself, as many times as it judges necessary. Between calls it reasons over what it has gathered and decides the next move — search again with a sharper query, search a different index, or stop and answer.

That unlocks a family of behaviors plain RAG cannot do: query rewriting, multi-hop retrieval, self-correction when results look thin, routing to the right source, and even skipping retrieval entirely when the question does not call for it. The model is no longer a passenger of your retrieval pipeline — it is steering.

The loop, in code

Underneath it is a tool-calling loop with a step cap: let the model search until it stops asking, then let it answer.

# Agentic RAG: the model calls "search" as a tool, as many times as it needs.
def agentic_rag(question, model, search, max_steps=5):
    messages = [{"role": "user", "content": question}]
    for _ in range(max_steps):                  # always cap the loop
        reply = model.chat(messages, tools=[SEARCH_TOOL])
        if not reply.tool_calls:                 # the model decided it has enough...
            return reply.content                 # ...and writes the final answer
        for call in reply.tool_calls:            # it may search, rewrite, then search again
            hits = search(call.args["query"])    # the model wrote this query itself
            messages.append(tool_result(call, hits))
    return model.chat(messages).content          # fallback answer once the step cap is hit

Everything sophisticated — re-ranking the hits, querying multiple indexes, grading retrieved context before continuing — is just richer logic inside this same loop. The core idea is the handoff: the model owns the decision of what to look up and when to stop.

Classic RAG retrieves and hopes. Agentic RAG retrieves, reads, and decides whether to retrieve again — which is exactly what a person does when one search is not enough.

The cost of the loop

• More calls, more latency. Each retrieval round is another model call; a multi-hop answer can take several, so it is slower and pricier than one-shot RAG.
• It can spin. Without a hard step cap and a clear stopping signal, a confused agent will search in circles — always bound the loop.
• Harder to debug. The retrieval path now varies per question, so you need tracing to see what the model searched and why it stopped.
• Overkill for simple apps. A well-tuned one-shot RAG still wins for FAQ-style lookups; reach for agentic RAG when questions are genuinely complex.
• Quality depends on the model. Driving retrieval well takes a capable model; a weak one writes poor queries and stops too early or too late.

The bottom line

Agentic RAG is the upgrade you reach for when single-shot retrieval keeps missing: multi-hop questions, ambiguous phrasing, multiple sources, or answers that need the model to look, think, and look again. Give search to the model as a tool, cap the loop, trace it, and you get retrieval that adapts to the question instead of betting everything on one query.

Do not reach for it by default — the extra calls cost real time and money, and plenty of apps are best served by fast, plain RAG. But when correctness depends on gathering the right evidence across several searches, letting the model decide what to retrieve is what closes the gap.