Model Cascades in Production: The Math That Decides If Routing Is Worth It

Every article on LLM cost tells you the same thing: most of your queries are easy, so send those to a cheap model and only escalate the hard ones to the expensive one. True, and useless -- it is advice, not engineering. The interesting question is not whether cascades save money. It is when they stop saving money, because they absolutely can cost you more than the naive approach, and the line between the two is a piece of arithmetic almost nobody writes down.

So let me write it down. A model cascade is not a free optimization. It is a bet with a known payoff structure, and if you know four numbers you can tell before you build it whether the bet pays. This post is about those four numbers, the break-even line they draw, and the three ways the whole thing quietly loses money in production even when the offline math looked great.

The mental model: a cascade is a bet on your escalation rate

Here is the frame that makes everything else fall out. In a cascade you always pay the cheap model. On the fraction of queries that escalate, you pay the cheap model AND the big model AND whatever the verifier costs to decide they should escalate. So a cascade only wins if the money you save on the queries that stay cheap is bigger than the money you burn double-paying on the queries that escalate.

That turns the whole decision into one variable you can actually measure: the escalation rate. Call it E -- the fraction of queries that fail the cheap model's check and get sent up. Everything hinges on E, and the mistake most teams make is building the cascade first and measuring E never.

You are not choosing between a cheap model and an expensive one. You are choosing between paying the big model's price on every query, or paying the cheap model's price on every query plus the big model's price on E of them plus a verifier tax on all of them. Write that sentence down before you write any code.

The break-even line, with actual arithmetic

Let me define the costs per query. Call the cheap model's cost c, the big model's cost b, and the verifier's cost v. In a single-model baseline you pay b on every query. In a cascade you pay c on every query, plus v on every query (you check everything), plus b on the E fraction that escalate. So the cascade wins when:

c + v + (E * b)  <  b

Rearranged to solve for the escalation rate E:

    E  <  (b - c - v) / b

That right-hand side is your break-even escalation rate.
Escalate MORE often than that and the cascade costs more
than just calling the big model on everything.

Now plug in illustrative numbers -- these are made up to show the shape of the math, not measured prices. Say the big model costs 10 units per query, the cheap model costs 1 unit, and your verifier is basically free because it is a length-and-keyword rule (v = 0). Break-even is (10 - 1 - 0) / 10 = 0.9. You can escalate up to 90% of queries and still come out ahead. That is the happy case everyone quotes.

Now make the verifier an LLM call -- say you use the big model itself to grade the cheap answer, so v = 10. Break-even becomes (10 - 1 - 10) / 10 = -0.1. It is negative. There is NO escalation rate that saves money, because your referee costs as much as the thing it is refereeing. This is the single most common way a cascade quietly loses: people pay a frontier model to check a frontier model.

One more, the realistic middle. Big model 10, cheap model 1, and a small verifier model at v = 0.5. Break-even is (10 - 1 - 0.5) / 10 = 0.85. If your real escalation rate is 30%, your cost per query is 1 + 0.5 + 0.3*10 = 4.5 units versus 10 for the baseline -- a 55% cut. If your escalation rate is actually 70% because the cheap model is weak on your traffic, it is 1 + 0.5 + 7 = 8.5 units, a measly 15% cut that may not survive the added latency and complexity.

Route up front, or escalate after -- a decision table

There are two families here and they have genuinely different economics. A router predicts difficulty before answering and commits to one model -- one call, no double-billing, but it pays for mistakes in quality because a mis-routed hard query gets a confident wrong answer with no second chance. A cascade tries cheap first and escalates on a bad answer -- it self-corrects, but it double-bills on escalations and adds a serial latency hop. Here is when each one is the right tool.

• Single model, no routing -- when your traffic is uniformly hard, or volume is low enough that engineering time costs more than the tokens you would save. Do not build a cascade to save forty dollars a month.
• Router (predict then commit) -- when a cheap, accurate difficulty signal exists (query length, endpoint, user tier, a small classifier) AND latency matters, because a router never adds a second serial call. Best when a wrong route is cheap to recover from downstream.
• Cascade (try cheap, verify, escalate) -- when you have NO reliable up-front difficulty signal but you CAN cheaply verify an answer after the fact, and correctness matters more than shaving the last few milliseconds. This is most RAG and Q&A traffic.
• Cascade with a non-LLM verifier -- the strongest case: when correctness is checkable by rules (does the SQL parse and run, does the JSON match the schema, does the number reconcile), because then v is near zero and your break-even escalation rate is close to 90%.

Notice the pattern: routers are gated by whether you can predict difficulty cheaply; cascades are gated by whether you can verify correctness cheaply. Pick the axis where you actually have a cheap signal. If you have neither, you do not have a routing problem, you have a single-model app, and that is fine.

A cascade in code -- and the one line that actually matters

The pattern is short. But the version you see in most blog posts forgets the thing that makes it maintainable: logging the escalation rate. Without that number you are flying blind on the exact variable that decides whether the whole system is worth running. Here it is with the instrumentation included.

import logging

metrics = {"total": 0, "escalated": 0}

def answer(query):
    metrics["total"] += 1

    # 1. Always try the cheap model first.
    draft = cheap_model.generate(query)

    # 2. Verify CHEAPLY. This must cost far less than big_model,
    #    or the whole cascade loses money (see the break-even math).
    #    Prefer a real check over asking a model "are you sure?".
    if is_good_enough(query, draft):
        return draft

    # 3. Escalate only the ones that failed the check.
    metrics["escalated"] += 1
    rate = metrics["escalated"] / metrics["total"]
    logging.info("cascade escalation_rate=%.3f", rate)

    return big_model.generate(query)


def is_good_enough(query, draft):
    # GOOD: structural / rule checks. v is ~0, break-even ~0.9.
    if not draft or len(draft) < 20:
        return False
    if "i am not sure" in draft.lower():
        return False
    # If your task is structured, verify the structure directly:
    #   return json_matches_schema(draft) and sql_runs(draft)
    return True

The steering wheel is the threshold buried inside is_good_enough. Loosen it and fewer queries escalate -- cheaper, riskier. Tighten it and more escalate -- safer, pricier. But the number to put on a dashboard is not the threshold, it is the escalation rate that threshold produces on live traffic, because that is the number in the break-even formula. Tune the threshold, watch the rate, compare the rate to your break-even line. That loop is the entire discipline.

Three ways it loses money in production. The offline math can look great and the thing still bleeds money once it is live. Three failure modes cause almost all of it, and none of them show up in a quick prototype.

The self-verification blind spot. If your verifier is the cheap model grading its own answer, it shares the cheap model's blind spots. On exactly the hard queries where it produced a confident wrong answer, it is also confident the answer is fine -- so it does not escalate, and the user gets the wrong answer at the cheap price. Self-grading catches the queries a model knows it struggled with and misses the ones it does not know it got wrong, which are the dangerous ones. Prefer an independent check: a rule, a different small model, or a retrieval-grounded verification.

The tail-latency trap. Average latency looks fine because most queries stop at the cheap model. But every escalated query pays cheap-model time PLUS verifier time PLUS big-model time, in series. So your p95 and p99 -- the numbers users actually feel -- get worse, not better, exactly on the hard queries that were already slow. If you have a latency SLA, budget it against the escalation path, not the average.

Silent drift. Your escalation rate is not a constant; it is a function of your traffic mix and both models' prices, and all three move. A new feature sends harder queries, E creeps from 30% to 65%, and your cost per query silently doubles while every individual call still looks normal. This is why the escalation rate belongs on a dashboard with an alert, not in a one-time notebook. The cascade that was a 55% win in June can be a 10% win in September and nobody notices until the bill arrives.

The bottom line

Cascades and routing are real wins, but they are wins with a break-even line, not free lunches. Before you build one, estimate three costs -- cheap model, big model, verifier -- and compute (b - c - v) / b to get the escalation rate you cannot exceed. After you build it, log the actual escalation rate and watch it drift. If your verifier is another frontier-model call, stop: the math is almost certainly negative and you are paying a genius to check a genius.

The mindset shift the generic version gets right is thinking in portfolios instead of a single best model. The part it leaves out is that a portfolio has a cost of management, and for a cascade that cost is the verifier plus the double-billing on escalations. Do the arithmetic first. The teams that win with cascades are not the ones with the cleverest router -- they are the ones who know their escalation rate to two decimal places and check it every week.