An LLM router picks the model and effort per request. How it works, how to choose the best router for AI agents, how to measure it, and when it does not pay.
Questions answered on this page
What is an LLM router?
An LLM router is a component that chooses which language model handles each request, instead of sending every request to one model you picked in advance. It sits between your application and the model providers, reads the incoming prompt, estimates how hard the task is, and dispatches it to a model that can do the job at the lowest price. The routing decision itself is usually a small trained classifier, not another language model call, because a router that asks a frontier model which model to use has already spent the money it was meant to save.
How does model routing work?
Model routing works in three steps, with an optional fourth. First, classify: a small model scores the prompt for difficulty and assigns it a tier. Second, select: the tier maps to a model in your configured pool, often together with a reasoning-effort setting. Third, execute: the selected model answers, and the request is logged with the model that actually ran. The optional fourth step is verification, where a separate scorer reads the completed answer and escalates the request to a stronger model when the answer misses a configured bar. Routers that stop after step two are prompt-side only. Routers that include step four are cascades.
How is an LLM router different from a fallback chain?
A fallback chain reacts to failure, a router decides on content. A fallback chain has an ordered list of models and moves to the next one when the current provider errors, rate-limits, or times out, so the model choice is driven by availability rather than by the prompt. A router picks a model because of what the prompt asks for, before anything has failed. Most production stacks want both: routing to decide the first attempt, fallback to keep the request served when that attempt cannot be made.
Does an LLM router make responses worse?
It can, and the rate at which it does is the number worth watching. Every router occasionally sends a request to a model that cannot do the job, which is called a catastrophic route. Cost savings are trivial to produce by sending everything to the cheapest model, so a savings figure with no catastrophic rate beside it says almost nothing. Evaluate a router on three numbers together: cost against a baseline you declared, served accuracy on outcome-labelled traffic, and the share of requests that landed on a model that could not handle them.
When is an LLM router not worth it?
Routing pays when your traffic is mixed in difficulty and you have some way to tell a good answer from a bad one. It pays little when traffic is uniformly hard, because there is no cheaper tier to move anything to. It pays little when the model is fixed by contract, by a compliance requirement, or by a fine-tune you depend on. It pays little when the bill is dominated by a large repeated input rather than by the per-token price of the model, where prompt caching and context trimming move more money than model choice does. And it is hard to justify on traffic with no measurable outcome, because without outcome labels you cannot separate a saving from a regression.
What is the best LLM router for AI agents?
There is no single best LLM router; the right one depends on where you want the decision to live. Pick by four criteria: whether it chooses the model and the reasoning effort together, whether it can run in shadow mode so you see its recommendations before it changes any traffic, whether it reports a catastrophic-route rate beside its savings figure, and whether it can sit beside the gateway you already run instead of replacing it. Nadir is built around those four: it recommends a model and effort per task, starts in shadow mode, writes a decision receipt per request, and works as a decision-only API or an optional OpenAI-compatible proxy. Routers that learn well on one benchmark can do worse on unseen agentic coding work, so test any shortlist on your own traffic before enforcing it.
How does a router choose both the model and the reasoning effort?
It treats effort as a second dimension of the same decision. The classifier scores the task, and the plan it returns names a model and an effort level together, because a cheaper model at higher effort can beat a pricier model at low effort on some tasks and lose on others. Nadir returns that pair as a recommendation first; you compare it with what your agent actually used, attach the outcome to the decision receipt, and enforce only for the routes that clear your quality bar.
How do you tell whether the verifier in a cascade is any good?
Score the verifier on held-out data the way you would score any classifier, and check its calibration, not just its ranking. A verifier that ranks answers well but is overconfident will escalate at the wrong threshold. Nadir's shipped reference-free verifier checkpoint scores held-out AUROC 0.776 with an expected calibration error of 0.028; the checkpoint and those figures are recorded in backend/app/settings.py. Reference-assisted evaluations, where the scorer is shown the expensive model's answer, produce much higher numbers and do not describe a deployed cost-saving cascade, which by definition does not have that answer.