Setup We ran 11,420 held-out triples from RouterBench through Nadir's verifier-gated cascade. The eval is public, the held-out split is disjoint from training (overlap_count=0), and the threshold sweep is reproducible from the open-source eval harness. Benchmark model: Claude Opus 4.6 (always-Opus baseline) Cheap tier: Claude Haiku 4.5; mid tier: Claude Sonnet 4.6 Pre-classifier: trained wide_deep tier classifier (under 10 ms per prompt) Verifier: calibrated outcome scorer (CPU INT8, 180 ms when it runs) Dataset: RouterBench held-out, n=11,420 triples The architecture Every prompt hits the pre-classifier first. If the pre-classifier is confident the prompt is cheap-tier, we skip the verifier entirely and ship the Haiku answer. That is the common path. If confidence is borderline, the cheap model answers first. The verifier reads that answer and scores it. If it accepts, we ship cheap. If it rejects, we escalate to Sonnet, or to Opus on Sonnet rejection. The verifier never sees Opus output; it only decides whether the cheap output is good enough to leave alone. The wedge against one-shot routers (Not Diamond, Martian) is the verification step. A predicted-cheap route can absorb a quality failure; a verified-cheap route surfaces it. Headline numbers | Metric | Value | |--------|-------| | Cost reduction vs always-Opus | 60% | | Quality preserved vs always-Opus | 98% | | Catastrophic routes | 1.7% | | Verifier AUROC | 0.961 | | Verifier calibration (ECE) | 0.016 | | Verifier latency, CPU INT8 | 180 ms | | Pre-classifier overhead | < 10 ms | "Quality preserved" is one minus catastrophic-route rate, on the same eval. We do not redefine the metric between sections. Threshold sweep The verifier threshold tau is the operational knob. Higher tau means the verifier rejects more cheap answers, escalating more often: better quality, less savings. Lower tau is the opposite. Sweep below is precomputed from the same eval. | tau | Accuracy | Cost reduction | Catastrophic | Wasted escalation | |-----|---------:|---------------:|-------------:|------------------:| | 0.3 | 88.2% | 73.2% | 8.8% | 3.0% | | 0.4 | 89.9% | 68.5% | 5.3% | 4.7% | | 0.5 | 90.3% | 67.0% | 4.4% | 5.3% | | 0.7 | 89.8% | 62.8% | 2.4% | 7.9% | | 0.8 | 89.2% | 60.9% | 1.7% | 9.2% | | 0.9 | 88.1% | 59.1% | 1.1% | 10.8% | The 60/98 headline numbers report tau=0.8. The shape of the curve matters more than the single operating point: cost reduction degrades gracefully as you tighten the quality guarantee. The cliff people fear is not there. How this beats prompt-only routing A prompt-only classifier sees only the input. It cannot tell you whether the cheap model handled it. The strongest prompt-only baseline on the same held-out split delivers 96.6% quality at 4.8x cost (where always-Opus is 12.0x and always-Haiku is 1.0x). The cascade hits 98.3% quality at 4.7x cost. Same cost, fewer quality drops, because reading the answer is cheaper than guessing whether it will be good. | Strategy | Cost (x) | Catastrophic | Quality preserved | |---|---:|---:|---:| | Always-Opus | 12.0x | 0% | 100% | | Prompt-only classifier | 4.8x | 3.4% | 96.6% | | Always-Haiku | 1.0x | 26.0% | 74.0% | | Verifier-gated cascade | 4.7x | 1.7% | 98.3% | What this means for your bill If you spend $5,000/month on LLM APIs and your prompt mix is what RouterBench held-out roughly reflects, Nadir saves about $3,000/month gross. With the 25% on first $2K / 10% above fee structure that is $600 in variable fees, $9 base, net about $2,391/month back. Complex prompts still hit Opus when the verifier says they need to. Reproducibility The eval harness, the held-out split, and the verifier weights ship with the image. Each release stamps a deterministic SHA on the classifier (visible per request as x-nadir-classifier-sha). Run the same triples; get the same numbers. The composed eval JSON is at verifier/reports/eval_composed_20260526T191001.json. The verifier calibration eval is at verifier/reports/eval_20260526T184516.json. RouterBench train/test disjointness is at verifier/reports/routerbench_contamination_20260524T122849.json.