LLM Routing Explained: How AI Model Routing Cuts Costs

The most expensive default in AI

The default behavior of every LLM SDK is to send every request to one model, the one you hard-coded, and for most teams that is the best and most expensive model available. It is the single most expensive way to run an LLM product, and it is what almost everyone does, because doing anything else requires a decision layer that the SDK does not ship.

That layer is LLM routing. This guide covers what it is, the architectures teams actually deploy, the one mistake that makes routing worse than not routing, and how to add it without rewriting your app.

What is LLM routing?

LLM routing is a decision layer that sits in front of multiple language models and, for each request, sends it to the cheapest model that can handle it well. AI model routing and LLM routing are the same idea under two names: instead of one model answering everything, an easy request goes to a small fast model and a hard request goes to a frontier model. You pay frontier prices only when the task actually needs them.

The economic case is simple and well measured. Meta's RouteLLM research found 74% of GPT-4-class calls did not need a GPT-4-class model to reach the same answer. If three of every four hard-model calls could have gone to a cheaper model at equal quality, then routing that traffic correctly is not a marginal optimization, it is most of the bill.

Why one model for everything is expensive

A production LLM workload is a mix. Some requests are classification, formatting, extraction, or short factual answers that a small model handles perfectly. Some are genuinely hard reasoning that needs the frontier. Sending the whole mix to the frontier model means the easy majority subsidizes nothing and simply costs more. Sending the whole mix to a cheap model saves money right up until the hard tail returns wrong answers and the quality complaints start.

The blanket swap is the trap. We tried replacing a frontier model with one a tenth the price across all traffic: it matched on 70% of prompts and broke badly on the hardest 30%, which is precisely the traffic a router is supposed to protect. Routing exists because neither extreme is right. The correct model is a per-request decision, not a per-app setting.

The five routing architectures

Teams deploy five patterns, often in combination:

• Static rules. Route by hard-coded signals: this endpoint uses the small model, that keyword triggers the large one. Cheap to build, brittle to maintain, blind to the actual difficulty of a request.
• Predictive classification. A classifier reads the prompt and predicts its difficulty, then maps that to a model tier. This is how Nadir's classifier works: a fast model estimates whether a request is simple, mid, or complex before any expensive call is made.
• Cascades. Try the cheap model first, and escalate to a more expensive one only if the answer looks low-confidence. This is the FrugalGPT pattern, and it is powerful because the decision to escalate is made after seeing an answer, not before.
• Semantic routing. Match the request against labeled example categories by embedding similarity, and route by category. Red Hat's vLLM Semantic Router is the notable open-source implementation, and it cut tokens 48% by routing simple queries out of always-on reasoning mode.
• Cache-aware routing. On self-hosted fleets, send requests that share context to the same replica so the KV cache is reused instead of recomputed. This is a latency and cost lever specific to running your own inference.

Most mature systems combine predictive classification with a cascade verifier: predict the tier, but confirm the cheap answer before trusting it.

How a verified router actually decides

The strongest routers add one component that separates them from the rest: a verifier. The flow is:

1. The router classifies the request and sends it to the cheapest model in the eligible tier.
2. That model returns an answer.
3. A calibrated verifier scores the answer against the request.
4. If the score clears a threshold, the answer ships. If not, the request escalates to a stronger model and the process repeats.

The reason this matters is that the router is now judged on the answer it actually produced, not on a guess it made before seeing one. That is the whole game. A router without step three is dead reckoning: it commits to a model based on a prediction, and when the prediction is wrong it has no way to know, so it ships the bad answer. Verification is what lets a router cut cost aggressively without cutting quality, because the only requests that get downgraded are the ones whose downgraded answer was checked and accepted.

A reference-assisted RouterBench research experiment on 11,420 triples projected 60% lower cost at about 98% retained quality. Because the verifier had the expensive reference answer unavailable in production, this is a research ceiling, not a deployed guarantee. Live quality requires outcome-labelled workload evidence.

Routing versus the alternatives

Routing is not the only way to cut LLM cost, but it is the one with the best ratio of savings to maintenance:

• Versus fine-tuning. Fine-tuning carries training, hosting, and re-tuning costs that recur every time a base model updates. Routing needs none of that and captures cheaper-model economics on the majority of traffic immediately.
• Versus caching. Caching and routing are complements, not substitutes. Caching removes the cost of repeated tokens; routing removes the cost of overpowered models. Run both.
• Versus spending caps. A cap limits request volume, not unit cost, and it fires after the bill is already incurred. Routing changes the price before the call.

If you want the full stack that routing sits inside, see the companion guide on how to reduce LLM API costs.

The other axis: what you send, not just who answers

Routing decides which model answers. Context selection decides what reaches it. They are separate levers on the same request, they multiply rather than overlap, and a router is the natural place to run both, because both are decisions made before a single output token exists.

Selection is query-conditioned pruning of retrieved context and tool output:

1. The latest user message is the query. It is never pruned, and neither is the system prompt.
2. Only large blocks are eligible. Retrieved context and tool output above roughly 800 characters. Parseable JSON is skipped so tool results stay loadable.
3. Each block splits into chunks at natural boundaries. Kept chunks survive byte for byte, nothing is rewritten or summarized.
4. Chunks are embedded and scored against the query. The encoder decides everything here: a lexical word-overlap scorer failed our quality gate at a 12.1% regression rate, and a semantic embedding model dropped that to 1.4%.
5. The top fraction survives, then the guards run. Lead and tail chunks always stay, safety and policy language is never dropped, and any chunk carrying a rare term from the question is pinned. That guard is why the answer paragraph survives even at tight budgets.
6. Dropped runs collapse into one marker line, so the model can see that something was omitted rather than silently missing it.

Measured on HotpotQA at a keep ratio of 0.6, every answer graded blind by an independent judge against the known-correct answer:

The routing-specific detail worth knowing: each block is scored against the question that follows it, not the newest one, so the decision is a pure function of the block and its own query and reproduces byte for byte as a conversation grows. That is what keeps a provider prompt cache hitting turn after turn, which matters more in a routed setup than an unrouted one, since a router that reshuffles the prefix on every turn quietly cancels the caching lever it is supposed to compose with.

The interactive walkthrough is on the context optimization page, the full method and its failure cases are in the benchmark writeup, and the algorithm is open source as barber-llm.

The one mistake: routing without verification

The failure mode worth naming twice is routing that never checks its own work. It is seductive because it is simple to build and it looks like it works: on easy traffic the cheap model is fine, the numbers drop, everyone is happy. Then a hard request gets misclassified as easy, the cheap model returns a confident wrong answer, and it ships, because there was no verifier to catch it. Unverified routing does not fail loudly. It fails as a slow quality regression that is hard to attribute back to the router. If you build or buy routing, the verifier is not optional.

How to add routing without a rewrite

The lowest-friction way to deploy routing is a gateway that speaks the OpenAI API format. You change the base URL, set the model to auto, and route per request without touching your call sites:

import openai

client = openai.OpenAI(
    base_url="https://api.getnadir.com/v1",
    api_key="ndr_...",
)

response = client.chat.completions.create(
    model="auto",  # router picks the cheapest capable model per request
    messages=[{"role": "user", "content": prompt}],
)

The request and response shapes are unchanged, so there is no SDK migration. Nadir reads the prompt, routes to the cheapest capable model, verifies the answer before returning it, and reports the measured savings against always-Opus per request. Bring your own provider keys and the router sits in front of them.

If you already run a gateway you like (LiteLLM, Kong, Envoy, Portkey), or your agent framework owns the provider call, you do not have to move the traffic at all. Nadir's decision API returns the routing decision on its own: POST /v1/bucket classifies the prompt, POST /v1/recommend returns the model pick against your own model list, in about 100 ms, and no prompt ever transits a model provider through Nadir. Your gateway keeps carrying the traffic. Nadir only answers which model the task deserves.

Conclusion

LLM routing is the layer the SDKs left out. It turns model choice from a per-app default into a per-request decision. A complete-response verifier can add evidence after generation, but it is not a universal quality guarantee: streaming bypasses that step, and customer quality still requires outcome-labelled traffic. The older 60% cost / 98% quality RouterBench result is a reference-assisted research ceiling, not deployed performance.

Related reading

• How to Reduce LLM API Costs by 60% Without Losing Quality
• RouteLLM: 74% of GPT-4 calls did not need GPT-4
• Routing without verification is dead reckoning
• FrugalGPT cascade routing: an implementation tutorial
• We tried to replace Opus with a model a tenth the price. Here is what broke.
• Can you speed up text? A third of input tokens removed, no answers lost

The 60% / ~98% figure is a reference-assisted research ceiling on 11,420 RouterBench triples against an always-top-model baseline. It is not deployed performance or a customer forecast.