Grounding Does Not Kill Hallucinations — It Just Moves Them Somewhere You Can See

The standard advice on LLM hallucinations fits on an index card: add retrieval so the model reads instead of remembers, tell it to say "I don't know," make it cite sources, and run a verifier. It is not wrong. It is just sold as a solution when it is really a relocation. Grounding does not remove the place where your system lies to a user. It moves that place out of the model's frozen weights and into your retriever, chunker, and verifier — components you own, can log, and can fix. That is a genuinely better spot to be in. But mistake "moved" for "eliminated" and you will ship it.

What follows is the version I wish I'd had before wiring grounding into anything real: a mental model that predicts where the new failures land, a worked example where grounding makes an answer more dangerous, a verification pattern with its own holes labeled, and a table for when not to reach for grounding at all.

The mental model: two different confidences, and the model only shows you one

Here is the frame that makes everything else click. A model has two distinct kinds of confidence, and they get collapsed into one voice on the way out. Recall confidence is how sure it is about something pulled from its own weights — smeared, dated, and impossible to calibrate, because "what I was trained on" is not a thing the model can introspect. Retrieval confidence is how well an answer is supported by text you just handed it in the prompt — and that one is checkable, because the evidence is sitting right there in the context window.

A hallucination is what happens when the model narrates low recall-confidence in the fluent, even tone of high confidence. The output layer has no dial that says "this next clause is a guess. " Grounding is powerful precisely because it swaps the unfixable kind (recall) for the fixable kind (retrieval). Once the fact is a quoted span in the prompt, "is this supported? " becomes a question you can actually answer with a second pass — instead of a question about the inside of a billion-parameter memory nobody can inspect.

Grounding's real trick is not making the model smarter. It is converting an unverifiable claim about memory into a verifiable claim about a document you control.

That reframing has a sharp consequence people skip: once you ground, your hallucination rate stops being a property of the model and starts being a property of your retrieval. A perfect model over a bad retrieval will confidently answer from the wrong paragraph. The failure just wears a different badge now.

A worked example: the support bot that grounding made worse

Say you run a support assistant over your product docs. A user asks: "Can I export my data after I cancel my plan? " Walk the two versions.

Before grounding (pure recall). The model answers from memory of a thousand SaaS products: "Yes, you can export your data for 30 days after cancellation. " Fluent, plausible, and completely made up for your product. Classic recall hallucination — nothing to check it against.

After grounding (naive RAG). You retrieve the top-3 doc chunks and instruct the model to answer only from them. The retriever pulls a chunk about exporting data — but it's the section for active accounts, because your docs never actually mention post-cancellation export, and that chunk was the nearest semantic neighbor. The model dutifully answers: "Yes, go to Settings and click Export. " It cited a real chunk. It followed your instructions. It is still wrong for the question asked — now with a citation stapled to it, which makes it more convincing to the user and harder for you to dismiss.

This is the failure mode the index-card advice hides: "grounded and cited" is not the same as "correct. " The retriever answered a subtly different question (how do I export? ) than the one asked (can I export after cancelling? ), and grounding gave that mismatch a veneer of authority. The fix is not more grounding. It is (a) letting the model refuse when the context does not address the specific question, and (b) verifying the answer against the specific claim, not just checking that some cited text exists.

The verification pattern — and where it leaks

The useful shape is: answer strictly from context, hand the model a real exit, then run a separate faithfulness check that is allowed to fail the whole thing. The verifier must be a fresh call with only the context and the answer — never the chat history — so it cannot be talked into agreeing by the same reasoning that produced the answer.

ANSWER_PROMPT = (
    "Answer ONLY from the CONTEXT below. If the context does not "
    "answer the exact question, reply with the single word: UNKNOWN. "
    "For every sentence, cite the chunk id you used like [c3].\n\n"
    "CONTEXT:\n{context}\n\nQUESTION: {question}"
)

# Verifier is a SEPARATE call. It never sees the question phrasing or
# the chat history that produced the answer -- only context + answer.
VERIFY_PROMPT = (
    "You are a fact-checker. For the ANSWER, decide if EVERY claim is "
    "directly supported by the CONTEXT. A claim that is plausible but "
    "not stated is NOT supported. Reply strict JSON with keys "
    "supported (bool) and unsupported_claims (list).\n\n"
    "CONTEXT:\n{context}\n\nANSWER:\n{answer}"
)

def grounded_answer(question, chunks):
    context = "\n".join("[" + c.id + "] " + c.text for c in chunks)
    answer = llm(ANSWER_PROMPT.format(context=context, question=question))

    if answer.strip() == "UNKNOWN":
        return {"reply": "I could not find that in our docs.", "safe": True}

    check = json_llm(VERIFY_PROMPT.format(context=context, answer=answer))
    if not check["supported"]:
        # Fail closed: an unverifiable answer is worse than no answer.
        return {"reply": "I could not confirm that in our docs.",
                "safe": True, "flagged": check["unsupported_claims"]}

    return {"reply": answer, "safe": True}

Read this as a scaffold, not a spell. It has real holes, and pretending otherwise is how the index-card version misleads:

• The verifier can hallucinate too. It is the same class of model doing the checking. It catches many unsupported claims but is not a proof — treat it as a smoke detector, not a fire marshal.
• It only checks faithfulness, not relevance. The cancellation example passes this verifier: the export steps ARE in the context. "Supported by context" does not mean "answers the question. " You need a separate relevance judgment for that.
• Strict JSON output is itself a hallucination surface. Parse defensively and treat a malformed verifier response as "not supported," or you have built a checker that fails open.

When to ground, when to skip it, and what breaks in production

Grounding is not free and not always right. It adds a retrieval hop, an extra model call for verification, latency, and a whole new class of retrieval bugs. Here is how I decide.

SITUATION                       GROUND?   WHY / WHAT BREAKS
-------------------------------------------------------------------
Facts about YOUR data,          YES       Recall cannot know your
docs, or a user account                   private data. Retrieval is
                                          the only honest source.

Fast-changing facts             YES       Weights are frozen at train
(prices, status, policy)                  time. Ground or be stale.

Open-ended reasoning,           NO        Nothing to retrieve. Grounding
brainstorming, rewriting                  adds latency + false
                                          "no source" refusals.

Well-known stable facts         MAYBE     Recall is usually fine; a bad
(capital cities, syntax)                  retriever can make it WORSE.

High-stakes / regulated         YES +     Ground AND verify AND log the
output                          verify    citations for audit. Fail
                                          closed, always.

Ultra-low-latency chat          CAUTION   Verification doubles calls.
                                          Consider verify-on-sample or
                                          only for risky intents.

The row people get wrong is "well-known stable facts. " Teams reflexively route everything through RAG, then watch the bot get a trivially known fact wrong because the retriever surfaced an unrelated chunk and the "answer only from context" instruction blocked the model from using the correct thing it already knew. Grounding a question the model would have nailed from recall can lower accuracy. Grounding is a scalpel, not a coat of paint.

And a handful of gotchas that never show up in the tutorials and always show up in your logs:

• Refusal rate is a KPI, not a failure. If you never see "I don't know" in production, your exit is decorative and the model is answering everything anyway. A healthy grounded system refuses a real slice of queries. As a rough rule of thumb, a refusal rate that is flat zero is a red flag, not a win.
• Chunk boundaries truncate facts. Retrieval that splits "you can export data — except after cancellation, when it is deleted" across two chunks and retrieves only the first will confidently tell users the opposite of the truth. The bug is in your splitter, not your model.
• Retrieval poisoning is grounding turned against you. If any retrieved corpus contains user-generated or crawled text, an attacker can plant a chunk that says "ignore prior instructions. " Grounding a hostile source is worse than no grounding, because you told the model to trust it.
• "Grounded but wrong" is the sneakiest state. A cited, faithful-to-context answer that is still incorrect (wrong chunk, stale doc, question mismatch) is harder to catch than a naked hallucination precisely because it looks accountable. Citations raise user trust whether or not the answer earns it.
• The verifier and the answerer sharing a model share blind spots. If both are the same model, a confident misreading of the context can pass its own check. Using a different model — or at least a fresh, history-free call — for verification buys you real independence.

The bottom line

Hallucination is not a bug you patch; it is a failure surface you relocate. Grounding moves the lie out of frozen weights and into your retrieval stack — a real upgrade, because now it is a plumbing problem you can log, test, and fix. But the corollary is the part the index card leaves off: your new job is retrieval quality, an honest exit, and a verifier you don't fully trust. Treat the model as a brilliant reasoner with an unreliable memory — never ask what it knows; give it what it needs, let it decline when you didn't, and check what it did with it. Do that, and "trust me" becomes "here is where it says so. " Skip the second half, and you've just built a fabricator with footnotes.