Speculative Decoding Is a Bet on How Predictable Your Traffic Is

Most explainers pitch speculative decoding as a free 2-3x speedup: flip a flag, get faster generation, lose no accuracy. All true, and all misleading. The framing hides the one number that decides whether you get 2.5x, 1.1x, or a regression that makes your p99 latency worse. That number is the acceptance rate, and it is not a property of the technique. It is a property of your traffic.

So here is the more useful mental model. Speculative decoding is a wager. On every step you spend extra compute drafting tokens you might throw away, betting the big model will agree with the guesses. When your text is predictable, you win the bet and cash in a batch of tokens for the price of one verification pass. When it is not, you eat the drafting cost and get little back. The technique does not make your model faster. It converts spare parallel compute into a lottery ticket whose payout is set by how guessable your output is.

Speculative decoding does not speed up your model. It sells you a lottery ticket priced in GPU cycles, and your traffic's predictability sets the odds.

The mental model: draft, verify, and the verification budget

The mechanism itself is genuinely simple, and it matters that you hold it in your head correctly. A small draft model generates the next k tokens serially - cheap, because it is small. Then the big target model runs a single forward pass that scores all k proposed positions at once. A transformer can score many tokens in parallel; only generation is serial. You accept the longest prefix of draft tokens the target agrees with, sample the first disagreement from the target itself, and discard the rest.

The accuracy guarantee is real and worth stating precisely: the accepted tokens follow exactly the target model's distribution, because the verification step is a rejection-sampling rule, not a similarity check. A bad draft model can never change what you output. It can only lower how often you win the bet. This is the one thing the technique gives you for free - correctness. Speed is the part you have to earn.

Now the piece the docs bury. Every speculative step costs you one target forward pass whether you accept zero draft tokens or all k. That pass is your verification budget, and you pay it every time. The whole game is amortizing that fixed cost across as many accepted tokens as possible. If you accept an average of 3 tokens per pass, you paid one pass to emit tokens that would have cost you roughly three passes without speculation. If you accept 0.5 tokens per pass, you paid a full pass plus the drafting overhead to emit less than one token - that is slower than plain decoding.

A worked example: when does the bet actually pay off?

Let me make this concrete with an illustrative back-of-envelope. These are made-up-but-plausible numbers to build intuition, not measurements - your real figures depend on your models and hardware, so measure before you trust any of this.

Say your target model takes 40 ms per forward pass, and your draft model takes 8 ms per token. You draft k=4 tokens ahead. Drafting costs 4 x 8 = 32 ms, and verification costs one 40 ms target pass, so a speculative step costs about 72 ms total. Compare that to plain decoding, where each token costs the full 40 ms target pass.

Now the payoff depends entirely on how many draft tokens you accept. Suppose on structured output - JSON, code, boilerplate prose - you accept 3 of the 4 drafts on average. You emit roughly 3 accepted plus 1 corrected token, so about 4 tokens for that 72 ms step. Plain decoding would have needed roughly 4 x 40 = 160 ms for the same 4 tokens. That is a bit over 2x. The bet paid off.

Now run the same math on high-entropy creative text where you accept just 0.5 drafts on average. You still pay the 72 ms step, but you only emit about 1.5 tokens. Plain decoding would have produced those 1.5 tokens in roughly 60 ms. You spent 72 ms to do what 60 ms would have done. You made it slower. Same flag, same models, opposite outcome - the only variable that changed was the acceptance rate, and that came from the prompt, not your config.

This is why 'does speculative decoding help? ' is an unanswerable question in the abstract. The honest version is: measure your acceptance rate on a representative sample of your actual traffic, then decide.

What it looks like in code - and where the sharp edges are

Serving stacks like vLLM, TensorRT-LLM, and llama. cpp ship this behind a flag, so you rarely write the loop yourself. But reading a stripped-down version tells you exactly which knobs exist and which ones bite. Here is one speculative step in pseudocode:

def speculative_step(target, draft, prefix, k):
    # 1. Draft k tokens serially (cheap, small model)
    draft_tokens, draft_probs = [], []
    ctx = prefix
    for _ in range(k):
        tok, p = draft.sample(ctx)
        draft_tokens.append(tok)
        draft_probs.append(p)
        ctx = ctx + [tok]

    # 2. ONE parallel target pass scores all k positions
    target_probs = target.score_parallel(prefix, draft_tokens)

    # 3. Accept prefix via rejection sampling, correct first miss
    accepted = []
    for i, tok in enumerate(draft_tokens):
        q = draft_probs[i][tok]        # draft's prob of this token
        p = target_probs[i][tok]       # target's prob of this token
        if random() < min(1.0, p / q):
            accepted.append(tok)       # keep it
        else:
            # reject: sample correction from the residual distribution
            corrected = sample_residual(target_probs[i], draft_probs[i])
            accepted.append(corrected)
            return accepted            # everything after a miss is dropped

    # all k accepted: sample one bonus token from the target
    accepted.append(target.sample_next(prefix + draft_tokens))
    return accepted

Three things in that snippet are the whole ballgame in production. First, the p over q ratio: the draft and target must share a tokenizer, or that ratio is meaningless and the guarantee breaks. Second, the early return on the first miss: one wrong token early throws away every correct token after it, which is why a draft that is right 90 percent of the time per token can still have a mediocre accepted-run length. Third, k is a tuning parameter with a peak, not a bigger-is-better dial - raise it too far and you spend more drafting time on tokens that get discarded after the first miss anyway.

A common simpler variant drops the separate draft model entirely and guesses using the target's own early layers, an n-gram lookup, or a small attached head (Medusa-style). That removes the second-model overhead and the tokenizer-mismatch risk in one move, at the cost of usually-lower acceptance. If two-model speculation is giving you tokenizer headaches, this is the first fallback to reach for.

The decision framework and the failure modes docs skip

Here is when the bet is worth placing, when it is not, and what actually breaks when teams turn it on in production:

• USE IT when your traffic is predictable - code generation, JSON and tool-call output, retrieval-grounded answers, templated or formal prose. High acceptance rates live here, and these are exactly the latency-sensitive interactive workloads where a 2x win matters.
• USE IT when you have idle parallel compute during serial decoding - typically low-to-moderate batch sizes on a single request or a few. The verification pass soaks up hardware you already paid for and were not using.
• SKIP IT for high-entropy generation - open-ended creative writing, brainstorming, poetry - where the next token is genuinely uncertain, acceptance craters, and you can go net-negative.
• SKIP IT at large batch sizes. When the GPU is already saturated serving many concurrent requests, there is no idle compute for verification to reclaim, and drafting steals throughput from real work. Speculative decoding optimizes latency, not throughput, and those two goals fight at scale.

The failure modes are where teams get surprised. Failure one: variance, not just averages. Speculative decoding makes your mean latency better and your latency variance worse. A hard prompt now has an unpredictable accepted-run length, so your p50 improves while your p99 can widen. If you have tight tail-latency SLOs, benchmark the tail, not the mean.

Failure two: the batch-interaction cliff. A setup that shines in your low-traffic staging tests can regress under production batch sizes, because the idle compute it relied on evaporates when the server fills up. The technique's benefit is inversely correlated with how busy your server is - the exact opposite of when you feel latency pain. Always benchmark at realistic concurrency, not on an idle box.

Failure three: silent drift when someone swaps the draft or target model without re-checking the pairing. A quantized draft, a fine-tune that shifted the distribution, or a tokenizer version bump can quietly halve your acceptance rate. Nothing errors - your outputs stay correct, they just get slower - so treat acceptance rate as a monitored production metric, not a one-time benchmark. If your speedup silently vanishes, this is the first place to look.

The bottom line

Speculative decoding earns its reputation, but 'free 2-3x speedup' is the wrong headline because it hides the work. The correct headline is: it converts spare parallel compute into speed at a rate your traffic sets, guarantees identical outputs while doing so, and gives back nothing - or worse than nothing - when your output is unpredictable or your server is saturated.

So do not think of it as a switch you flip. Think of it as a bet you size. Pick a draft model with a shared tokenizer, tune k, and then do the one thing that actually decides the outcome: measure acceptance rate on your real traffic, at your real batch sizes, watching your real tail latency. The technique gives you correctness for free. Speed is the part you have to go win.