Mixture of Experts, from a Capacity-Planning Chair: The Router Is Cheap, the VRAM Bill Is Not

There are two numbers on the spec sheet of a modern open model, and most explanations of Mixture of Experts spend all their energy on the wrong one. They celebrate the active-parameter count — the 30B or 40B that actually fires per token — because it sounds like a free lunch. Fine. But if you are the person who has to decide whether this model fits on the box you are renting, the active count is not the number that hurts you. The total count is.

So this is not another walk through top-k routing for its own sake. It is the mental model I reach for when someone drops a sparse model in my lap and asks 'will this run, and how fast. ' The one-sentence version: an MoE model has two budgets that live in two different places, and MoE's whole value proposition is that it lets you spend them separately.

Active parameters price your compute and your latency. Total parameters price your memory and your knowledge. Every honest MoE decision starts by reading those two numbers as two different bills.

The mental model: a call center, not a brain

Forget 'the model only switches on part of itself. ' That framing makes people imagine memory shrinking, and it doesn't. Picture a call center. You employ 128 specialists (the experts). Every one is on payroll, sitting at a desk, taking up floor space — that is your rent, and you pay it whether or not they get a call. When a ticket comes in (a token), a dispatcher (the router) reads it and forwards it to the 2 best-suited specialists. Only those 2 do work on that ticket.

Two consequences fall straight out of the metaphor. Your electricity bill — work actually done — scales with how many specialists you dispatch per ticket, which is tiny. Your rent — desks you must keep — scales with how many you employ, which is huge. A dense model is a call center where every ticket is handled by the entire staff at once: enormous electricity bill, same rent. MoE keeps the rent and slashes the electricity. Hold that picture, because it predicts every gotcha later in this post without you memorizing any of them.

Stated as an equation you can plan with: in a dense transformer every token flows through every parameter of each feed-forward block, so capacity and compute are welded — double the params, double the per-token FLOPs, forever. MoE unwelds them by replacing that one big block with N experts plus a router that keeps the top k. Total params scale with N (your memory footprint and stored knowledge). Active params scale with k (your compute per token, hence latency and throughput). The router itself is a rounding error: one small linear layer producing one score per expert.

A worked capacity calculation (the part explainers skip)

Say you are handed a sparse model advertised as '128 experts, top-2, 30B active, 240B total' and told to serve it. The active number tempts you toward a single 48GB card. Don't sign anything yet. Walk the memory math — these are illustrative round numbers to show the shape of the calculation, not a spec for any specific model.

1. Weights: 240B params. At FP16 that is 2 bytes each — about 480GB. Even experts that never fire this second must stay resident, because the next token might route to them. That alone overflows any single GPU you can rent casually.
2. Quantize to 4-bit and you are near 120GB — now it is two 80GB cards or a small multi-GPU node, plus overhead.
3. KV cache: sized by active compute and context length, not total params. This part behaves like a normal ~30B-class model. Comfortable.
4. The punchline: your latency budget says 'this is a 30B model,' your memory budget says 'provision for 240B. ' Size the box off the active number and you are now explaining an out-of-memory crash in a postmortem.

That split — cheap to run, expensive to hold — is the single most load-bearing fact about serving MoE, and it is exactly the fact the 'runs like a 40B model' headline buries.

The router, load balancing, and what the code really says

The router is a small learned layer that, per token, scores every expert and keeps the top k. During training the experts drift toward loose specialities — some lean into code, some into a language's morphology, some into punctuation-level syntax — rarely in ways you can cleanly label. The model learns the experts and the routing to them at once, and because the decision is discrete (you pick experts, you don't smoothly average all of them), it is fragile in a specific way.

Back to the call center: if the dispatcher decides two specialists are 'safe' and routes everything to them, the other 126 never get tickets, never improve, and never earn their rent. This is expert collapse, and it silently vaporizes capacity you paid VRAM for. So MoE training adds an auxiliary load-balancing loss that pushes tokens to spread evenly — one of the rare places in deep learning where you deliberately fight the model's own gradient preference. Here is the forward pass in the shape it actually takes; production kernels batch tokens per expert so the GPU never stalls on a per-expert loop, but the logic is identical.

import torch
import torch.nn.functional as F

def moe_layer(x, experts, router, k=2):
    # x: (num_tokens, d_model)
    scores = router(x)                      # (num_tokens, num_experts)
    topk_val, topk_idx = scores.topk(k, dim=-1)
    weights = F.softmax(topk_val, dim=-1)   # normalize the chosen k only

    out = torch.zeros_like(x)
    for slot in range(k):
        idx = topk_idx[:, slot]             # which expert each token picked
        w = weights[:, slot].unsqueeze(-1)
        for e in range(len(experts)):
            mask = idx == e
            if mask.any():
                out[mask] += w[mask] * experts[e](x[mask])
    return out

# The router is trivial to compute. The cost you actually pay is that
# every experts[e] must already be resident in memory, all num_experts
# of them, even though only k contribute to any given token.

Read the last comment twice. The loop makes it visually obvious that only k experts contribute — and just as obvious that the whole list has to exist in memory for the indexing to work. The metaphor, the equation, and the code all say the same thing.

When to reach for MoE — and when it will just hurt

Sparsity is a tool, not a virtue. Here is the decision table I use before recommending an MoE model over a dense one of comparable active size.

• USE IT when you are memory-rich but compute-constrained — VRAM to hold a big model, but you need low latency per token. This is the home-run case.
• USE IT when traffic is broad and varied (many domains, many languages) so the breadth parked in idle experts actually gets exercised.
• AVOID IT when VRAM is your bottleneck — a dense model of the active size gives comparable per-parameter quality for a fraction of the memory. Don't pay 240B of rent to do 30B of work when you are short on desks.
• AVOID IT when traffic is narrow and repetitive — you are housing 126 specialists to answer the same two kinds of ticket, which a small dense model does for less.
• BE CAUTIOUS when fine-tuning — routing can shift under a new dataset, so small MoE fine-tunes sometimes behave less predictably than dense ones. Budget extra evaluation.

Notice the asymmetry that trips people up: an MoE with 30B active is usually NOT as strong, per active parameter, as a dense 30B. Its edge is entirely the knowledge parked in the experts it did not run this token. If your workload never provokes that parked knowledge, you bought rent you never use.

Gotchas in production, and the number to carry into the meeting

None of these are exotic. Each falls straight out of the call-center picture, which is the whole reason I lead with the metaphor.

• Memory does not shrink with sparsity. Every expert loads even though few fire — provision VRAM for total params, always. This is the mistake I see most.
• Batching scatters your tokens to different experts, so efficient serving needs expert-parallel scheduling dense engines never think about. Throughput hinges on how well your inference engine handles it.
• A hot expert becomes a straggler. If real traffic over-routes one expert, that GPU shard is the bottleneck and tail latency spikes even though average FLOPs look fine. Watch per-expert utilization, not just aggregate.
• Load balancing is a live loss term, not a setup step. Tune it wrong and experts collapse or starve, discarding paid-for capacity with no crash to alert you.

MoE decouples what a model knows from what it costs to run each token — you scale knowledge by employing more experts and scale compute by how many you dispatch. Genuinely powerful, and why sparse is the default now. But 'runs like a 40B model' is a claim about your electricity bill only; your rent is set by the total count, and rent is due whether or not the desks are busy.

So when the next sparse model lands, do the boring thing first: write both numbers on the whiteboard, put 'latency and throughput' under the active one and 'VRAM and knowledge' under the total one, then size the box off the second while you quote the first. Understanding the router is part of understanding the model — but understanding which number pays which bill is the part that keeps you out of a postmortem.