Fine-Tuning, RAG, or Prompting? Diagnose the Failure Mode, Not the Vibe

There are three ways to make a large language model do what you want, and roughly a thousand blog posts telling you which to pick. Almost all of them give you the same tidy answer: RAG for knowledge, fine-tuning for behavior, prompting first. That's not wrong. It's just not a decision procedure. It's a slogan you can nod along to and still get badly wrong in practice.

I want to give you the thing the slogan skips: a way to look at a bad answer your model just produced and diagnose which lever will actually fix it — before you spend a quarter and a GPU budget finding out the hard way.

Stop choosing a technique. Start reading the failure.

The single biggest mistake I see is that teams pick a technique first and a problem second. Someone read that fine-tuning is powerful, so they fine-tune — to add facts, which is exactly what fine-tuning is bad at. Someone else bolts on a vector database to fix a model that keeps answering in the wrong tone, which retrieval cannot touch. The technique was chosen for its reputation, not for the specific way the model was failing.

Flip it. Treat the model's wrong output like a bug report. A good engineer doesn't reach for a debugger, a profiler, and a rewrite all at once — they reproduce the failure and localize it first. Same here: get a handful of real wrong answers in front of you and ask a colder question than 'which tool' — what, precisely, is wrong with this specific response?

RAG changes what the model knows. Fine-tuning changes how it behaves. Prompting changes what you asked. Most wasted quarters come from using one to do another one's job.

Four failure buckets, not two

The usual framing is a binary: knowledge gap versus behavior gap. That's a good start but it's too coarse — it lumps together failures that need different fixes. I split it into four buckets, because in real triage the fourth one is where most of the confusion actually lives.

• Missing knowledge — the answer depends on facts the model never saw (your internal docs) or facts that changed after training (today's pricing, this week's incident). Fluent, confident, wrong. Fix: RAG.
• Wrong behavior — the model knows the answer but delivers it in the wrong shape: chatty when you need terse, prose when you need JSON, hedging when you need a verdict. Fix: fine-tuning, once prompting stops holding.
• Under-specified ask — the model did what you literally said, which wasn't what you meant. There is no gap in the model at all; the gap is in your prompt. Fix: prompting. This is the most common failure and the cheapest, and people skip straight past it.
• Capability ceiling — the task genuinely exceeds what the model can do at any prompt (multi-step reasoning it fumbles, a domain it has no grip on). No lever fixes this cleanly; you change models, decompose the task, or add tools.

The reason four beats two: if you only have 'knowledge vs behavior,' every under-specified-ask failure gets misfiled as one or the other, and you go build infrastructure for a problem that a better sentence in your prompt would have solved. Naming the fourth bucket is what stops you from over-engineering.

A worked example: the support bot that keeps making things up

Say you run a support assistant for a SaaS product. Users complain it gives confidently wrong answers about your refund policy, and separately, that its replies are three paragraphs of corporate mush when people just want the one-line answer. Here's the trap: the two complaints feel like one problem ('the bot is bad'), so teams reach for one fix.

Read the failures separately. The refund-policy errors are a missing-knowledge failure — your policy lives in a Notion page the model never trained on, and it changed last month. That's RAG: retrieve the current policy text and answer from it. Fine-tuning the policy into the weights would be the classic expensive mistake; policies change, weights don't, and you'd be back here next quarter — plus fine-tuning facts tends to make hallucination worse, not better, because the model learns the shape of confident answers without a reliable source to anchor them.

The three-paragraphs-of-mush complaint is a different bucket. First, don't assume it's a behavior gap — check bucket four's neighbor, the under-specified ask. Did anyone actually tell the model to be terse? Often the 'fix' is a single line in the system prompt: answer in one or two sentences, lead with the direct answer, no preamble. Try that before you fine-tune anything. If, after a genuinely good prompt, it still drifts back to verbose on 20% of calls, that's when the behavior gap is real and fine-tuning earns its place — you bake terse-and-direct into the model so you don't have to re-litigate it in every prompt.

Notice what happened: one 'bad bot' split into a knowledge fix (RAG) and a behavior fix (prompt first, fine-tune only if it plateaus), and one of those fixes was almost free. That split is the whole game. Here's the same triage compressed into a function — the value isn't the code, it's the order of the questions: cheapest, most-reversible check first, heavy infrastructure last.

def triage(bad_answer):
    # 1. Cheapest question first: did we actually ask for the right thing?
    if not clearly_specified(bad_answer.prompt):
        return 'PROMPT'          # fix the instructions, re-test in seconds

    # 2. Is the content wrong because facts are missing or stale?
    if depends_on_private_or_fresh_facts(bad_answer):
        return 'RAG'             # retrieve the facts; do NOT train them in

    # 3. Content is fine, but the shape/tone/format is consistently off
    #    AND a good prompt already failed to hold it?
    if behavior_wrong and prompt_only_fix_plateaued(bad_answer):
        return 'FINE_TUNE'       # bake the behavior into the weights

    # 4. None of the above -> the task exceeds the model itself
    return 'CHANGE_MODEL_OR_DECOMPOSE'

The guardrail hiding in that function is the plateau check on step three. Fine-tuning is the only lever here with real, ongoing cost — a data pipeline, a training run, an eval harness, and a version you now own forever. You should have to prove prompting failed before you're allowed to reach for it.

The order in that function is a cost order, and the asymmetry is the whole point. The numbers below are illustrative rules of thumb, not measurements — orders of magnitude to reason with. Prompting is reversible in seconds and costs a few extra tokens per call; anyone can do it. RAG is days to a couple of weeks of engineering, plus an index to keep fresh and retrieval latency on every call. Fine-tuning is weeks — the labeled dataset is usually the real bottleneck — plus re-training whenever your base model or requirements move, plus an eval suite you now own forever.

That asymmetry isn't a reason to avoid fine-tuning. It's a reason to make it prove it's necessary, and to spend that cost only on behavior that's genuinely stable and genuinely worth freezing.

The gotchas nobody puts in the tidy diagram

These are the failure modes that show up after you ship, when the neat knowledge-vs-behavior story meets production:

• Stale-index rot. RAG is only as fresh as your last re-index. If the refund policy changes and your pipeline re-embeds nightly, you ship yesterday's answer with today's confidence. RAG moves the freshness problem; it doesn't delete it. Own the re-index cadence explicitly.
• The eval gap on fine-tuning. It's easy to fine-tune and feel great from eyeballing ten outputs. Without a held-out eval set you cannot tell whether you improved the model or just overfit to your training examples — and you won't find out until users do.
• Fine-tune lock-in across model upgrades. A base model two generations newer, prompted well, often beats your lovingly fine-tuned older model. But your fine-tune is welded to that old base. Every fine-tune is a small bet against your own future upgrade path — price that in.
• Retrieval poisoning by your own docs. RAG faithfully retrieves whatever's in the corpus, including the outdated wiki page, the contradictory draft, and the intern's wrong note. Retrieval doesn't adjudicate truth; it surfaces text. Curate the corpus like it's production data, because it is.
• Prompt-bloat latency creep. The 'free' lever isn't free at scale. Every few-shot example and instruction rides along on every single call, inflating both latency and token cost. A prompt that grew organically to twenty examples can quietly become your biggest line item — and sometimes that's the actual signal that this behavior should be fine-tuned in instead.

The bottom line

Don't argue about fine-tuning versus RAG versus prompting in the abstract — the abstract argument has no answer. Put a real wrong answer in front of you and diagnose it: under-specified ask (fix the prompt), missing knowledge (RAG), stable wrong behavior that prompting can't hold (fine-tune), or a genuine capability ceiling (change the model). Start with the cheapest, most reversible lever and make each heavier one prove it's needed.

The mature systems don't pick one. They prompt well, retrieve for knowledge, and fine-tune the handful of behaviors worth freezing — each lever doing only the job it's actually good at. But that layered system is the destination, not the starting move. The starting move is always the same: stop choosing a technique, and start reading the failure.