What did Nadir's RouterBench cascade benchmark measure?

It ran 11,420 held-out RouterBench triples through a cascade: a trained pre-classifier sent confident prompts straight to a tier, and borderline prompts went to Claude Haiku 4.5 first, with a verifier deciding whether to escalate. The baseline was always using Claude Opus 4.6. The held-out split had zero overlap with training.

Is the 60% cost reduction a production number?

No. It is a reference-assisted research ceiling; the verifier had the expensive reference answer that production does not have. The 60% lower cost at 98% quality preserved, with 1.7% catastrophic routes at tau 0.8, is not deployed performance, a customer forecast, or a billing baseline.

How does the verifier threshold trade cost against quality?

In the reference-assisted sweep, lowering tau from 0.8 to 0.3 raised cost reduction from 60.9% to 73.2% but raised catastrophic routes from 1.7% to 8.8%. Raising it to 0.9 cut catastrophic routes to 1.1% at 59.1% cost reduction. Production needs its own reference-free calibration on outcome-labelled traffic.

Did checking the answer beat prompt-only routing in this experiment?

At roughly matched cost, the verifier-gated cascade had 1.7% catastrophic routes against 3.4% for a prompt-only classifier. That shows reference-assisted output evidence helped. It does not show that the deployed reference-free verifier closes the same gap.

Reference-assisted RouterBench cascade: the 60% / 98% research…

We ran 11,420 held-out RouterBench triples through a reference-assisted cascade whose verifier had the expensive reference answer. That answer is unavailable to the deployed reference-free path; AUROC 0.961, ECE 0.016, 60% projected savings, and 98% retained quality are therefore a research ceiling, not production performance. Here are the raw numbers and the boundary that matters.

Evidence correction, 27 August 2026: this was a reference-assisted experiment. The verifier received the expensive-model reference answer that the deployed reference-free cascade does not have. The numbers below are a research ceiling, not production performance, a customer forecast, or a billing baseline. The raw reports are local eval artifacts and are not reproducible from a clean clone today.

Setup

We ran 11,420 held-out RouterBench triples through a reference-assisted cascade. The held-out split was disjoint from training (overlap_count=0), but the verifier input did not match the live reference-free path.

Benchmark model: Claude Opus 4.6 (always-Opus baseline)
Cheap tier: Claude Haiku 4.5; mid tier: Claude Sonnet 4.6
Pre-classifier: trained wide_deep tier classifier (under 10 ms per prompt)
Verifier: reference-assisted outcome scorer; the expensive answer was supplied
Dataset: RouterBench held-out, n=11,420 triples

The architecture

Every prompt hit the pre-classifier first. High-confidence cases skipped the verifier, while borderline cases entered the experimental cascade.

If confidence was borderline, the cheap model answered first. In this experiment the verifier scored that answer together with the expensive reference answer. That reference made the result an oracle-like research ceiling; a cost-saving live cascade cannot obtain it without paying for the expensive call first.

The experiment showed why output evidence can outperform prompt-only prediction. It did not establish that the deployed reference-free verifier closes the same gap.

Research-ceiling numbers

Metric	Value
Cost reduction vs always-Opus	60%
Quality preserved vs always-Opus	98%
Catastrophic routes	1.7%
Verifier AUROC	0.961
Verifier calibration (ECE)	0.016
Verifier latency, CPU INT8	180 ms
Pre-classifier overhead	< 10 ms

"Quality preserved" is one minus catastrophic-route rate in this reference-assisted eval. It is not deployed served quality.

Threshold sweep

The verifier threshold tau was the experimental knob. The sweep below is precomputed from the same reference-assisted eval and must not be used to configure production.

tau	Accuracy	Cost reduction	Catastrophic	Wasted escalation
0.3	88.2%	73.2%	8.8%	3.0%
0.4	89.9%	68.5%	5.3%	4.7%
0.5	90.3%	67.0%	4.4%	5.3%
0.7	89.8%	62.8%	2.4%	7.9%
0.8	89.2%	60.9%	1.7%	9.2%
0.9	88.1%	59.1%	1.1%	10.8%

The 60/98 research ceiling reports tau=0.8. Production needs a separate reference-free calibration on the customer's model ladder and outcome-labelled traffic.

What the experiment says about prompt-only routing

A prompt-only classifier sees only the input. The table shows that reference-assisted output evidence improved the held-out curve at matched projected cost. It does not show that today's reference-free production verifier realizes that improvement.

Strategy	Cost (x)	Catastrophic	Quality preserved
Always-Opus	12.0x	0%	100%
Prompt-only classifier	4.8x	3.4%	96.6%
Always-Haiku	1.0x	26.0%	74.0%
Verifier-gated cascade	4.7x	1.7%	98.3%

What this means for your bill

Nothing universal. Start with the model or policy the customer actually runs, shadow Nadir's decisions over measured traffic, attach outcomes, and price only savings the managed proxy executes. Decision-only and shadow counterfactuals remain advisory.

Reproducibility

The local harness and artifacts produced this result, but verifier/reports/ is gitignored and the full evidence bundle is not available in a clean clone. Do not describe this result as publicly reproducible until the report, hashes, split manifest, and exact weights are published together.

The composed eval JSON is at verifier/reports/eval_composed_20260526T191001.json. The verifier calibration eval is at verifier/reports/eval_20260526T184516.json. RouterBench train/test disjointness is at verifier/reports/routerbench_contamination_20260524T122849.json.

More on routing & cascades

What Nadir is

Nadir is an LLM router. Nadir sizes every prompt and routes it to the cheapest model that still clears your quality bar. A small decision model reads each prompt against your model menu and picks the lowest-cost model it expects to handle it. When the decision model is uncertain or unavailable, the request goes to the strongest model on the menu. The routing step makes no call to your model provider.

Nadir runs two ways. The decision API returns a model, reasoning-effort, cache, context, and policy recommendation without calling a model provider, beside the gateway you already run. That is how a shadow-mode evaluation works, and its projected savings stay advisory. The OpenAI compatible managed proxy executes the route, and migration is a two-line change: point the base URL at api.getnadir.com and set model to auto. On that path an optional verifier can score a complete non-streaming answer and escalate to a stronger model when it misses the configured bar. Streaming bypasses post-generation verification. BYOK is supported on every tier.

For coding agents, Nadir Auto is a terminal launcher for Claude Code or Codex that asks Nadir for the main-session model before each turn, with your configured model as the fallback and cost ceiling. Delegation integrations for Claude Code, Codex, or Cursor instead recommend a model tier for subagent work, and the agent decides; the Claude plugin can also move a main-thread turn to Haiku when Nadir's cache check prices it cheaper, unless its switch_main_model option is off. Either way inference runs on your own provider account, so Nadir holds no provider keys on that path.

What the numbers are, and what they are not

Nadir publishes each evaluation with its scope. On checkable code, run-check-escalate solved 392 of 395 common HumanEval and MBPP problems (99.2%), graded by running the canonical tests; that applies only to tasks with runnable deterministic tests. Nadir-Tumbler posts an arena_score of 72.3 on RouterArena's public scorer, 5th of 23 routers, which measures the routing decision on RouterArena's own model pool. A reference-assisted RouterBench evaluation over 11,420 held-out triples produced a 60% lower projected cost than always-Opus with about 98% retained quality and a 1.7% catastrophic-route rate. That experiment gave the verifier the expensive-model reference answer, which production does not have, so it is a research ceiling and not the deployed path.

None of these is a production guarantee, a universal savings rate, or a forecast for any particular workload. Customer savings are reported from measured execution against a declared baseline, and customer quality only from outcome-labelled traffic. Projected savings and realized savings are separate artifacts and are never blended.

Design-partner program

Three rungs, picked by risk appetite. Rung 0 Shadow runs advisory decision calls alongside live traffic and returns a projected receipt, with nothing in the request path changed. Rung 1 Hosted is the two-line swap on a production slice and returns a realized receipt. Rung 2 On-prem is a supervised six-week proof of concept inside the partner's VPC, where no prompt, response, or usage reaches Nadir. The commitments are the same at every rung. Apply for a rung directly: Rung 0 Shadow, Rung 1 Hosted, or Rung 2 On-prem. Not sure which fits? Start at getnadir.com/contact/?reason=design-partner.

Licensing

NadirClaw is the self-hosted core, source-available under the PolyForm Noncommercial License. Source-available is the correct label; NadirClaw is not open source. Nadir Route's hosted plan has no base fee and charges a variable fee only on measured savings from requests Nadir executed.

Reference-assisted RouterBench cascade: the 60% / 98% research ceiling