Speculative Decoding: How to Make LLM Inference 2–3× Faster

Large language models are slow for a boring reason: they generate one token at a time. Each token needs a full forward pass through the whole model, and you cannot start token N+1 until token N is done. On a big model, that serial dependency — not raw compute — is what you feel as latency.

Speculative decoding breaks that bottleneck without retraining anything and without changing a single output. It routinely makes generation 2–3× faster while producing exactly the text the big model would have produced on its own. It is one of the highest-leverage tricks in modern inference, and the idea is surprisingly simple.

The idea: draft, then verify

The insight is that verifying a guess is much cheaper than generating from scratch. A transformer can score many tokens in parallel in a single forward pass — it is only generation that is serial. So you let a small, fast model do the guessing and the big model do the checking.

• A small draft model quickly proposes the next few tokens — say four or five — one after another. It is cheap, so this is fast.
• The big target model runs ONE forward pass over all of those proposed tokens at once, scoring what it would have predicted at each position.
• You accept the longest run of draft tokens that matches the target's own choices, then sample the first mismatch from the target. Everything after the mismatch is thrown away.
• Repeat. On an easy stretch of text you might accept four tokens for the price of one big-model pass; on a hard one you fall back to roughly normal speed.

Why it is free accuracy

The part that surprises people: the output is mathematically identical to plain decoding from the big model. The verification step uses a rejection-sampling rule that guarantees the accepted tokens follow exactly the target model's distribution. The draft model can be dumb, biased, or wrong — it only ever proposes; the target always has the final say.

That means there is no quality trade-off to argue about. A bad draft model just lowers your acceptance rate (less speedup); it can never change what you generate. The only thing you are tuning is speed.

What it looks like in code

Most serving stacks — vLLM, TensorRT-LLM, llama. cpp — ship this behind a flag, but the loop is short enough to read. Stripped down, one speculative step looks like this:

# One speculative step: draft k tokens, verify in a single target pass.
def speculative_step(prefix, draft, target, k=4):
    # 1) Draft model proposes k tokens, cheaply and serially.
    proposed = draft.generate(prefix, max_new_tokens=k)

    # 2) Target scores the whole proposed block in ONE forward pass.
    target_logits = target.forward(prefix + proposed)        # parallel over positions

    # 3) Accept the longest prefix of proposals the target agrees with.
    accepted = []
    for i, tok in enumerate(proposed):
        if accept(tok, target_logits[i]):                    # rejection-sampling rule
            accepted.append(tok)
        else:
            accepted.append(sample(target_logits[i]))        # fix the first mismatch
            break
    return accepted   # 1..k tokens, distributed exactly like plain target decoding

Everything else — picking k, choosing a draft model, batching — is tuning around that core. A common, even simpler variant skips the separate draft model entirely and uses the model's own earlier layers or n-gram lookups to guess, which removes the cost of running two models.

Speculative decoding is the rare optimization with no downside in quality — the worst case is that it gives you nothing, and the typical case is that it doubles your speed.

When it helps, and when it does not

• Best on predictable text. Code, structured output, and formal prose have long stretches the draft model nails, so acceptance rates — and speedups — are high.
• Weaker on creative or high-entropy text, where the next token is genuinely uncertain and the draft is often wrong.
• Needs a good draft model: same tokenizer, much smaller, and ideally distilled from or trained alongside the target. A mismatched draft kills your acceptance rate.
• Loves spare compute. The verification pass uses parallel hardware you were not fully using during serial decoding, which is why latency drops without extra GPUs.
• Less useful at very large batch sizes, where the GPU is already saturated and there is no idle compute for verification to soak up.

The bottom line

If you serve an LLM and care about latency, speculative decoding is close to a free lunch: a real 2–3× speedup, identical outputs, and a flag away in most modern serving engines. The main work is choosing and maybe distilling a good draft model, and measuring acceptance rates on your actual traffic.

It is a good reminder that a lot of inference speed is not about bigger GPUs — it is about not wasting the parallel compute you already paid for. Draft, verify, keep what matches, and let the hardware do in one pass what used to take four.