Continuous Batching, Read Through One Number: Average GPU Seat Occupancy

Almost every explanation of continuous batching stops in the same place: it is a scheduler that swaps requests in and out of the batch every token instead of every request, so the GPU stays busy. That is correct, and it is the least useful version of the idea, because it tells you what the code does without telling you what to watch when your serving bill is three times what you budgeted.

Here is the version I actually use when reasoning about an LLM deployment. It centers on one number you can put on a dashboard, and it turns the failure modes from surprises into things you can predict.

The one number: how full the batch is, averaged over time

The mental model: a GPU serving a language model is a bus with a fixed number of seats. Driving the bus one block costs almost the same whether it carries one passenger or fifty. So your cost per passenger is decided by one thing only, averaged across the whole trip: how many seats were occupied. Throughput, cost per token, the marketing multipliers quoted for vLLM versus a naive loop, all of them are downstream of that average. Every batching strategy is just a different answer to: how do we keep more seats filled, more of the time?

Static batching keeps the bus parked at every stop until the last passenger from the previous group climbs off. Continuous batching lets people board and leave a bus that never stops rolling. Same bus, same fuel, radically different cost per rider.

Framed this way, static batching's flaw is concrete. A batch of 32 where 31 finish at token 40 and one runs to token 800 spends its last 760 steps carrying a single passenger on a 32-seat bus. Peak occupancy was perfect; average occupancy is dismal. Continuous batching attacks exactly that gap between peak and average.

A worked before/after on one GPU

A concrete scenario. These numbers are illustrative, chosen to make the arithmetic clean, not measured, but the shape is real. Say you have one GPU that holds 32 concurrent sequences, requests arrive steadily, and output lengths are mixed: most replies are short (around 50 tokens), a few long (around 800).

With static batching, you gather 32 requests, run them together, and cannot return any or admit new ones until the slowest finishes. If one runs to 800 tokens, all 32 seats stay reserved for 800 steps. The short requests finished near step 50 but their seats sit empty for 750 steps while new arrivals queue outside the door. Picture occupied seats over time as a bar chart: it starts at 32, decays fast toward 1, then snaps back to 32 for the next group. The area under that curve, your real utilization, is a fraction of the rectangle you paid for.

With continuous batching, the moment a short request emits its final token at step 50, its seat frees and a queued request boards on step 51. The long request keeps going, now surrounded by a rotating cast of short requests filling the other 31 seats continuously. The occupancy curve stops sawtoothing and flattens near the ceiling. You did not buy a faster GPU; you stopped paying for empty seats. The takeaway: the win scales with how variable your output lengths are. Uniform replies? Static batching is already near-optimal. Wildly mixed lengths, which is what real chat and agent traffic look like? That is where the multiplier lives.

Two machines sharing one GPU: prefill and decode

This is the part tidy explanations skip, and it is where production surprises come from. A running request does two very different kinds of work, and continuous batching juggles both:

• Prefill: processing the prompt you sent. It chews through hundreds or thousands of input tokens in one compute-heavy pass. It is compute-bound and can briefly hog the GPU.
• Decode: generating output one token at a time. Each step touches the whole model to produce a single token, so it is memory-bandwidth-bound, not compute-bound. This is the loop continuous batching keeps full.

The non-obvious consequence: when a new request arrives, its expensive prefill step can stall the smooth decode loop of everyone already generating. Naive schedulers create a stutter where latency for existing users spikes every time a long prompt joins. That is why serving engines expose chunked prefill, which slices a giant prompt into pieces so it interleaves with ongoing decode instead of blocking it. If per-token latency jumps whenever traffic spikes, prefill contention is your first suspect. And because decode is memory-bandwidth-bound, adding sequences is nearly free on compute but not free on the KV cache each one carries, which is the real ceiling.

The real ceiling is memory, and why PagedAttention exists

Continuous batching can only fill seats that exist, and the seat count is set by how many KV caches fit in GPU memory at once. Each request's KV cache grows with every token it generates. The old way reserved the maximum length up front and contiguously, so a request that might reach 2048 tokens booked memory for 2048 while it was still on token 12. Multiply that across 32 requests and the bus loses most of its seats to reservations nobody is using.

PagedAttention, vLLM's contribution, treats the KV cache like operating-system virtual memory: small non-contiguous pages, handed out only as a request needs them. Reclaimed memory becomes more seats, and more seats is exactly the fuel continuous batching needs to stay full. The two are a matched pair, the scheduler and the memory system that gives it enough seats to schedule. The gotcha that follows: because pages fill as sequences grow, a batch that fit at the start can run out of KV memory mid-flight. Engines handle this by preempting, evicting a request's cache and recomputing it later. Preemption is invisible until it is not; under memory pressure it shows up as latency cliffs. If tail latency degrades only under load, look at preemption and swap counters, not at the model.

The scheduler loop, in honest pseudocode

Stripped to essentials, the loop admits and retires requests around one shared forward pass. Note that admission is gated on both a sequence-count limit and a memory budget, because those are two different ceilings:

def serve_step(running, waiting, kv_budget, max_seqs):
    # Admit waiting requests into free seats
    while waiting and len(running) < max_seqs:
        req = waiting[0]
        needed = req.estimate_kv_pages()
        if needed > kv_budget:
            break                 # no memory: leave it queued
        kv_budget -= needed
        running.append(waiting.pop(0))

    # One shared forward pass: every running seq emits one token
    step_batch(running)

    # Retire finished requests; free their seats and memory
    for req in list(running):
        if req.is_done():
            kv_budget += req.release_kv_pages()
            running.remove(req)
            req.return_to_user()

    return running, waiting, kv_budget

When to reach for it, and when not to

Continuous batching is the default for good reason, but treating it as a universal win produces over-tuned configs and blown latency SLAs. A quick decision list:

• Reach for it: many concurrent users, mixed and unpredictable output lengths, throughput-first or shared multi-tenant endpoints. This is the sweet spot and the source of the big multipliers.
• It barely helps: single-user or very low concurrency. With one request in flight there are no empty seats to fill, so you pay a serving engine's fixed cost for little of its upside.
• It can hurt if misconfigured: strict low-latency SLAs. Cranking max concurrent sequences for throughput raises per-token latency for everyone, since each forward pass now carries more work. Throughput and latency trade against each other; you cannot max both.
• Wrong tool entirely: offline batch jobs where you control all inputs up front. Plain large static batches with inputs sorted by length are simpler and just as efficient there.

The knobs that matter are the two ceilings from the loop: max concurrent sequences (seat count) and max batched tokens (per-step work budget, which governs how much prefill and decode share one pass). A minimal, illustrative starting point:

# Illustrative vLLM-style starting point, not a universal recommendation.
# Tune to YOUR traffic; watch occupancy and p99 latency together.
max_num_seqs: 64            # seat count: raise for throughput, watch latency
max_num_batched_tokens: 8192 # per-step budget shared by prefill + decode
enable_chunked_prefill: true # slice big prompts so they do not stall decode
gpu_memory_utilization: 0.90 # leave headroom; too high invites preemption

The discipline: change one knob, watch two graphs, average batch occupancy and p99 per-token latency, and stop when raising concurrency stops improving occupancy or starts breaking your latency budget. That crossover is your optimum, and it is specific to your traffic. Nobody can hand you the number.

The bottom line

You will almost never implement continuous batching yourself; vLLM, TGI, and TensorRT-LLM already have it. But knowing it is fundamentally a fight to keep average seat count high changes how you run a deployment. Feed the engine many concurrent requests instead of trickling them one at a time. Expect a large win when output lengths vary and a small one when they do not. Treat prefill contention and KV-memory preemption as the two prime suspects when latency misbehaves. And tune the two ceilings against your own occupancy and tail-latency graphs rather than copying someone else's config. That is where cost per token actually drops, and it is a number you can watch fall.