Execution-graded code results, offline agent replay, independent router scoring, and a reference-assisted research ceiling separate from deployed performance.
Questions answered on this page
What does each Nadir benchmark measure?
Each result comes from a different execution mode. Checkable code is graded by running tests. The SWE-bench figure is an offline replay over published per-instance results. RouterArena is an independent scorer of routing decisions. RouterBench is a reference-assisted research ceiling. None of them forecasts a customer's bill.
How was the checkable-code result graded?
By executing each problem's canonical tests, with no LLM judge. The cheapest model answers first and a failed test escalates to Claude Sonnet 5, then Claude Opus 5. That solved 392 of 395 common HumanEval and MBPP problems, against 383 for Claude Opus 5 on its own. It applies only to work whose output can be checked by running it.
Is the 39.8% SWE-bench saving a leaderboard result?
No. It is an offline replay of a tier policy over published per-instance SWE-bench results: 371 of 500 tasks resolved against 384 for always Claude 4.5 Opus High, at 39.8% lower replay cost. It is not a live SWE-bench submission and does not use the deployed classifier.
Where does Nadir rank on RouterArena?
nadir-caliper ranked #9 of 33 routers with an arena_score of 74.63, 76.14% accuracy, $0.25 per 1K queries and robustness 79.76, checked 2026-11-01. Measures routing decisions on RouterArena's model pool; does not predict customer savings or served quality.
Why is the RouterBench result called a research ceiling?
Because the verifier in that experiment could see the expensive model's answer while it scored the cheap one. A production cascade that saves money never has that answer, so the deployed verifier works reference-free and scores lower. Treat the RouterBench figures as an upper bound, not as deployed accuracy or a savings forecast.
How much will Nadir save on my own traffic?
Nadir does not quote a universal number. Start in shadow mode, where Nadir records the model it would have picked without changing any response, then compare that against your declared baseline on your own requests before you let it route.