LLM Routing and Model Cascades: Cut Cost Without Cutting Quality

Pick your best model, point every request at it, ship. It is the default way to build an LLM app, and it works — it is also how you end up paying frontier prices to answer questions a model a tenth the size could have handled. You are using a sledgehammer for every nail.

Routing and cascades are the fix. The idea is to match each query to the cheapest model that can actually handle it, instead of sending everything to the most expensive one. Done well, it cuts cost by half or more with barely any drop in quality — one of the highest-return changes you can make to a production LLM system.

Most queries are easy

Look at real traffic and the pattern is stark: a large share of requests are simple — a greeting, a lookup, a reformat, a short factual answer — and a small share are genuinely hard, needing deep reasoning or long context. Yet a one-model app charges the hard-query rate for every single one.

• Easy queries dominate. Most of what users ask does not need your strongest model to get a correct answer.
• Hard queries are rare but real. A minority truly need the big model, and sending those to a small one produces confident, wrong answers.
• Price gaps are huge. The strongest models can cost ten to fifty times more per token than a capable small one, so the model mix decides your bill.
• Paying the top rate for everything is the waste — the goal is to pay it only when a query actually earns it.

Two patterns: route up front, or escalate

There are two ways to spend less. A router predicts difficulty before answering — a small classifier (or even a length and keyword heuristic) decides whether a query goes to the small or the large model. One decision, one call, but you are betting the router guesses right.

A cascade is more forgiving: always try the cheap model first, then check whether its answer is good enough — by its own confidence, a quick verifier, or simple rules — and escalate to the bigger model only when it is not. It costs an extra call on the hard cases, but it almost never sends a hard query to a model that cannot handle it.

A cascade, in code

The whole pattern fits in a few lines: answer cheaply, keep it if the model is sure, escalate if it is not.

# Cascade: try the cheap model first, escalate to the expensive one only if unsure.
def answer(question):
    draft = small_model.complete(question, logprobs=True)
    if confidence(draft) >= 0.8:               # small model is confident -> done, ~10x cheaper
        return draft.text
    return big_model.complete(question).text   # genuinely hard -> pay for the strong model

# confidence() can read the model's own logprobs, ask it to self-rate, or run a
# cheap verifier; the threshold is the dial between cost savings and quality.

Tune the threshold and you are directly trading money for quality: raise it and more queries escalate (safer, pricier); lower it and more stay cheap (cheaper, riskier). That one number is the steering wheel.

You do not need your smartest model to say good morning. Routing is just refusing to pay genius prices for questions a junior could answer.

Getting it right, and where it bites

• Pick a routing signal. A small trained classifier, an embedding similarity, query features, or the cheap model's own confidence all work — start simple and measure.
• Mis-routing is the real risk. Sending a hard query to the small model yields a fluent wrong answer, which is worse than a slow right one. Bias the threshold toward escalating when unsure.
• Cascades add latency on escalation. When the cheap model fails, you pay two calls and the wait of both — fine on average, but watch the tail.
• Verify cheaply. The accept-or-escalate check has to be much cheaper than the big model, or you have spent your savings on the referee.
• It drifts. Query mixes and model prices change, so revisit the routing logic and re-measure cost and quality on real traffic periodically.

The bottom line

Routing and cascades are the cheapest big lever in production LLM cost. Send easy queries to a small model, escalate the hard ones to a large model, and tune the threshold until cost drops without quality following. For most apps that is a 50% to 80% bill cut for a few lines of glue code.

The mindset shift is to stop thinking in terms of one model and start thinking in terms of a portfolio. Your job is not to pick the best model — it is to send each request to the cheapest one that still gets it right.