Continuous Batching: How LLM Servers Hit High Throughput

If you serve one model to many users, your bill is decided by throughput: how many requests one GPU can handle at once. And the difference between a server that handles ten concurrent users and one that handles two hundred on the same hardware is rarely the model — it is how the requests are batched.

Continuous batching is the scheduling technique that makes the difference. It is the quiet engine behind vLLM, TGI, and TensorRT-LLM, and it is the reason modern serving stacks get several times the throughput of a naive setup. Understanding it explains most of why one deployment is cheap and another is not.

Why static batching wastes the GPU

The obvious way to batch is to collect a group of requests, run them together, and return them together. It works, but it leaves the GPU idle for much of the time, because generation is a loop and requests do not finish in step with one another.

• Requests have wildly different lengths. One reply is 10 tokens, another is 800 — but a static batch cannot return the short one until the long one is done.
• Finished slots sit empty. As shorter requests complete, their place in the batch goes idle while the batch waits for the slowest member.
• New requests have to wait. Anyone who arrives mid-batch is stuck in a queue until the whole current batch drains, even though the GPU has free capacity.
• The result is low utilization — you paid for a GPU that spends much of its time waiting rather than computing.

Batching per token, not per request

Continuous batching — also called in-flight or iteration-level batching — moves the scheduling decision down to a single step of generation. Every model produces one token per request per forward pass, so the scheduler reconsiders the batch at each of those steps instead of once per request.

That changes everything. The moment a request emits its final token, it leaves the batch and returns to the user immediately — and the moment a slot opens, a waiting request joins mid-flight. The batch is constantly reshaped so the GPU is always working on as many live requests as it can hold.

The scheduler, in pseudocode

Stripped to its essence, the loop admits and retires requests around a single shared forward pass:

# Continuous batching: schedule per token, not per request.
running = []                                  # requests currently generating
while running or queue:
    while queue and can_fit(running):         # admit new requests as soon as slots free up
        running.append(queue.pop())           # ...mid-flight, not after a whole batch drains
    logits = model.step(running)              # ONE forward pass over all running sequences
    for req in list(running):
        req.append(sample(logits[req]))       # each request advances by one token
        if req.is_done():
            emit(req)                          # finished requests leave the instant they end,
            running.remove(req)                # freeing their slot for the next waiting one

No request waits for an unrelated one to finish, and the batch never drains to empty between groups. That single change is most of the throughput win.

Static batching asks every request to start and stop together. Continuous batching lets them come and go like passengers on a bus that never stops moving — and the GPU is the bus you are paying for by the hour.

The other half: PagedAttention and the KV cache

• The memory bottleneck. Each running request keeps a KV cache that grows as it generates; reserving the maximum length up front, contiguously, wastes huge amounts of GPU memory.
• PagedAttention fixes it. vLLM stores the KV cache in small non-contiguous pages, like an operating system's virtual memory, so you only use what each request actually needs.
• Less waste means more seats. Reclaimed memory lets you keep far more requests resident at once, which is exactly what continuous batching needs to stay full.
• Tune the limits. Knobs like max concurrent sequences and max batched tokens trade per-request latency against total throughput — set them for your traffic.
• You rarely build this yourself. vLLM, TGI, and TensorRT-LLM implement both continuous batching and paged KV caching; the win is choosing and configuring them, not reimplementing them.

The bottom line

Continuous batching turns an LLM server from a polite queue into a packed, constantly-reshuffled batch that keeps the GPU busy. Paired with a paged KV cache, it is why a serving engine can quietly deliver several times the throughput of a hand-rolled loop — on the exact same hardware.

You almost never implement it yourself, but knowing what it does changes how you deploy: reach for a real serving engine, feed it many concurrent requests rather than one at a time, and tune the batch limits to your traffic. That is where the cost-per-token actually drops.