Mixture of Experts: Why Frontier Models Route Instead of Compute

Read the spec sheet of a recent frontier model and you will see two parameter counts. One is enormous — say 744 billion. The other, the one labelled active, is a fraction of it — maybe 30 or 40 billion. The model holds a vast amount of knowledge but only switches on a small part of itself for any given token. That gap is Mixture of Experts (MoE), and it is now the default way to scale.

The promise is almost too good: keep adding capacity — more facts, more skills — without paying more compute per token. A well-built MoE gives you the knowledge of a giant dense model at the inference cost of a medium one. The catch is that the savings live entirely in a tiny, temperamental component called the router.

Dense versus sparse

In a normal (dense) transformer, every token flows through every parameter of each feed-forward layer. Double the parameters and you double the compute for every token, forever. MoE replaces that one big feed-forward block with many smaller ones — the experts — and a router that sends each token to just a couple of them.

• A dense layer: one feed-forward network, used by every token. Capacity and compute are locked together.
• An MoE layer: many feed-forward experts plus a router. Only the chosen experts run for a given token.
• Top-k routing: the router scores all experts and picks the best k — often just 2 of 8, 16, or more. The rest stay idle for that token.
• Result: total parameters scale with the number of experts, but compute scales only with k. You buy capacity and pay for a slice.

The router is the whole trick

The router is a small learned layer that, for each token, produces a score per expert and keeps the top few. Different experts drift toward different specialities during training — syntax, code, particular languages — though not in ways humans can cleanly label. The model learns both the experts and how to route to them at the same time.

Because routing is learned and discrete, it is also fragile. If the router falls in love with a handful of experts, the rest never train and your giant parameter count is wasted. So MoE training adds a load-balancing pressure that nudges tokens to spread across experts evenly — one of the few places you deliberately fight the model's instincts.

What it looks like in code

The forward pass is shorter than the theory suggests. Score the experts, keep the top k, run only those, and blend their outputs by the router's weights:

# One MoE layer: route each token to its top-k experts, then blend.
import torch
import torch.nn.functional as F

def moe_layer(x, experts, router, k=2):
    scores = router(x)                       # (tokens, num_experts)
    weight, idx = scores.topk(k, dim=-1)     # pick the best k experts per token
    weight = F.softmax(weight, dim=-1)       # normalize the k gate weights

    out = torch.zeros_like(x)
    for slot in range(k):
        for e in range(len(experts)):
            mask = idx[:, slot] == e          # tokens routed to expert e in this slot
            if mask.any():
                out[mask] += weight[mask, slot, None] * experts[e](x[mask])
    return out   # only k experts ran per token, not all of them

Real implementations batch this far more cleverly so the GPU is not stalled by per-expert loops, but the shape is the same: a cheap routing decision, then a few expert calls instead of one giant one.

Mixture of Experts decouples what a model knows from what it costs to run — you scale knowledge by adding experts and scale cost only by how many you switch on.

The trade-offs nobody mentions

• Memory does not shrink. Every expert has to be loaded even though only a few run, so a sparse 744B model still needs the VRAM of a 744B model — you save compute, not storage.
• Serving is harder. Tokens in a batch scatter to different experts, so efficient MoE inference needs careful expert-parallel scheduling that dense models never worry about.
• Load balancing is a real loss term. Get it wrong and experts collapse or starve, quietly throwing away the capacity you paid for.
• Fine-tuning is touchier. Routing can shift under a new dataset, so small MoE fine-tunes sometimes behave less predictably than dense ones.
• Quality per active parameter is lower. An MoE with 40B active params is usually not as strong as a dense 40B — its edge is the knowledge stored in the experts it did not run.

Why it matters for builders

If you serve open models, MoE changes your math. Budget VRAM for the total parameter count, not the active one, and expect throughput to depend on how well your engine handles expert parallelism. The headline active-parameter number tells you about compute and latency; the total number tells you about memory and knowledge.

MoE is not magic — it is a bet that most tokens only need a sliver of a model's knowledge, and that a small router can pick the right sliver. So far that bet keeps paying off, which is why nearly every new frontier model is sparse. Understanding the router is now part of understanding the model.