The most expensive default in AI The default behavior of every LLM SDK is to send every request to one model, the one you hard-coded, and for most teams that is the best and most expensive model available. It is the single most expensive way to run an LLM product, and it is what almost everyone does, because doing anything else requires a decision layer that the SDK does not ship. That layer is LLM routing. This guide covers what it is, the architectures teams actually deploy, the one mistake that makes routing worse than not routing, and how to add it without rewriting your app. What is LLM routing? LLM routing is a decision layer that sits in front of multiple language models and, for each request, sends it to the cheapest model that can handle it well. AI model routing and LLM routing are the same idea under two names: instead of one model answering everything, an easy request goes to a small fast model and a hard request goes to a frontier model. You pay frontier prices only when the task actually needs them. The economic case is simple and well measured. Meta's RouteLLM research found 74% of GPT-4-class calls did not need a GPT-4-class model to reach the same answer. If three of every four hard-model calls could have gone to a cheaper model at equal quality, then routing that traffic correctly is not a marginal optimization, it is most of the bill. Why one model for everything is expensive A production LLM workload is a mix. Some requests are classification, formatting, extraction, or short factual answers that a small model handles perfectly. Some are genuinely hard reasoning that needs the frontier. Sending the whole mix to the frontier model means the easy majority subsidizes nothing and simply costs more. Sending the whole mix to a cheap model saves money right up until the hard tail returns wrong answers and the quality complaints start. The blanket swap is the trap. We tried replacing a frontier model with one a tenth the price across all traffic: it matched on 70% of prompts and broke badly on the hardest 30%, which is precisely the traffic a router is supposed to protect. Routing exists because neither extreme is right. The correct model is a per-request decision, not a per-app setting. The five routing architectures Teams deploy five patterns, often in combination: Static rules. Route by hard-coded signals: this endpoint uses the small model, that keyword triggers the large one. Cheap to build, brittle to maintain, blind to the actual difficulty of a request. Predictive classification. A classifier reads the prompt and predicts its difficulty, then maps that to a model tier. This is how Nadir's classifier works: a fast model estimates whether a request is simple, mid, or complex before any expensive call is made. Cascades. Try the cheap model first, and escalate to a more expensive one only if the answer looks low-confidence. This is the FrugalGPT pattern, and it is powerful because the decision to escalate is made after seeing an answer, not before. Semantic routing. Match the request against labeled example categories by embedding similarity, and route by category. Red Hat's vLLM Semantic Router is the notable open-source implementation, and it cut tokens 48% by routing simple queries out of always-on reasoning mode. Cache-aware routing. On self-hosted fleets, send requests that share context to the same replica so the KV cache is reused instead of recomputed. This is a latency and cost lever specific to running your own inference. Most mature systems combine predictive classification with a cascade verifier: predict the tier, but confirm the cheap answer before trusting it. How a verified router actually decides The strongest routers add one component that separates them from the rest: a verifier. The flow is: The router classifies the request and sends it to the cheapest model in the eligible tier. That model returns an answer. A calibrated verifier scores the answer against the request. If the score clears a threshold, the answer ships. If not, the request escalates to a stronger model and the process repeats. The reason this matters is that the router is now judged on the answer it actually produced, not on a guess it made before seeing one. That is the whole game. A router without step three is dead reckoning: it commits to a model based on a prediction, and when the prediction is wrong it has no way to know, so it ships the bad answer. Verification is what lets a router cut cost aggressively without cutting quality, because the only requests that get downgraded are the ones whose downgraded answer was checked and accepted. On 11,420 RouterBench held-out triples, a verified router cut cost 60% versus always using the top model while preserving about 98% of that model's quality. Take away the verifier and you can still cut cost, but you can no longer promise the quality, because nothing checked the cheap answers. Routing versus the alternatives Routing is not the only way to cut LLM cost, but it is the one with the best ratio of savings to maintenance: Versus fine-tuning. Fine-tuning carries training, hosting, and re-tuning costs that recur every time a base model updates. Routing needs none of that and captures cheaper-model economics on the majority of traffic immediately. Versus caching. Caching and routing are complements, not substitutes. Caching removes the cost of repeated tokens; routing removes the cost of overpowered models. Run both. Versus spending caps. A cap limits request volume, not unit cost, and it fires after the bill is already incurred. Routing changes the price before the call. If you want the full stack that routing sits inside, see the companion guide on how to reduce LLM API costs. The one mistake: routing without verification The failure mode worth naming twice is routing that never checks its own work. It is seductive because it is simple to build and it looks like it works: on easy traffic the cheap model is fine, the numbers drop, everyone is happy. Then a hard request gets misclassified as easy, the cheap model returns a confident wrong answer, and it ships, because there was no verifier to catch it. Unverified routing does not fail loudly. It fails as a slow quality regression that is hard to attribute back to the router. If you build or buy routing, the verifier is not optional. How to add routing without a rewrite The lowest-friction way to deploy routing is a gateway that speaks the OpenAI API format. You change the base URL, set the model to auto, and route per request without touching your call sites: import openai client = openai.OpenAI( base_url="https://api.getnadir.com/v1", api_key="ndr_...", ) response = client.chat.completions.create( model="auto", # router picks the cheapest capable model per request messages=[{"role": "user", "content": prompt}], ) The request and response shapes are unchanged, so there is no SDK migration. Nadir reads the prompt, routes to the cheapest capable model, verifies the answer before returning it, and reports the measured savings against always-Opus per request. Bring your own provider keys and the router sits in front of them. Conclusion LLM routing is the layer the SDKs left out. It turns model choice from a per-app default into a per-request decision, and when it is verified, it captures the savings of cheaper models on the easy majority of traffic without giving up frontier quality on the hard tail. The architectures vary, but the one that matters is the one with a verifier in the loop, because that is the only version that lets you cut cost 60% and still stand behind the answer. Everything else is a guess with a discount attached. Related reading How to Reduce LLM API Costs by 60% Without Losing Quality RouteLLM: 74% of GPT-4 calls did not need GPT-4 Routing without verification is dead reckoning FrugalGPT cascade routing: an implementation tutorial We tried to replace Opus with a model a tenth the price. Here is what broke. The 60% cost reduction with ~98% quality retention figure is measured on 11,420 RouterBench held-out triples against an always-top-model baseline. Routing outcomes vary with your traffic mix; the harder your workload skews, the more the verifier earns its place.