Shadow mode catches the regression. It also stops the moment you flip to 100%.
A cheaper model that matches frontier quality shows up every few weeks now: MiniMax M3 undercutting Opus 4.8, DeepSeek V4 at a fraction of Anthropic and OpenAI's rates, Meta's Muse Spark opening at a quarter of the going output price. The economics are obvious. So is the fear: swap the model under a production agent and the failures don't show up in a demo, they show up three weeks later on a customer's edge-case prompt. Read what happened when one team tried the blanket swap.
The industry's answer, borrowed straight from SRE, is a staged rollout. Duplicate live traffic to the candidate model without showing anyone its output, score the pairs offline, then promote gradually with an automated rollback trigger watching every stage. It's the right instinct. It also has a blind spot nobody talks about: the moment the rollout finishes, so does the checking.
What "shadow" and "canary" actually mean
Shadow mode mirrors production requests to both the incumbent model, which answers the user, and the candidate, which doesn't. An automated judge scores the two outputs against each other on accuracy, tone, task completion, and format compliance, while the team also tracks token count, cost, and latency deltas. Because every request gets answered twice, shadow mode roughly doubles inference spend for as long as it runs, usually one to two weeks. Source: TianPan, "Releasing AI Features Without Breaking Production," April 2026
Canary deployment is the next stage: the candidate serves real users, starting at 1% of traffic, sometimes 0.1% for anything high-stakes, and ramping 1% → 5% → 20% → 50% → 100%, holding each stage at least 24 hours before advancing. Source: TianPan, "Releasing AI Features Without Breaking Production," April 2026 Mirror sampling, running shadow at 10-25% of full traffic instead of 100%, cuts the overhead while still producing enough scored pairs to catch a distribution-level regression: a 10% mirror on a million daily requests generates roughly 100,000 scored pairs every 24 hours. Source: Future AGI, "LLM Eval with Shadow Traffic and Canary Deployment," 2026
Both stages lean on automated rollback triggers, because a human watching a dashboard doesn't catch a regression fast enough. The common set: guardrail trip rate above 1.5x baseline in a 15-minute window, an offline rubric score that regresses below the noise floor, p99 latency above 1.3x baseline, or an error cluster that shows up only on the candidate and wasn't present in the incumbent's trailing week. Median rollback latency once one of those trips: about 35 seconds. Source: Future AGI, "LLM Eval with Shadow Traffic and Canary Deployment," 2026
Skip the discipline and the failure is predictable. One team that flipped a candidate model to 100% of traffic without staged validation saw function-call accuracy drop 9 points and refusal rate climb 7 points, both invisible until the flip was already live. Source: Future AGI, "LLM Eval with Shadow Traffic and Canary Deployment," 2026
The gate that closes after you walk through it
Here's the part the runbooks don't emphasize: shadow and canary validate the swap, not the traffic. They answer "is this candidate model good enough to trust with my production workload," once, over a sample of requests gathered during a specific one-to-two-week window. Once the candidate hits 100% and the pipeline is torn down, nothing is comparing its answers to anything anymore. It's just the model now.
That's fine if your traffic tomorrow looks like your traffic during the rollout. It rarely does. A new customer in a market your shadow sample under-represented, a longer context than anything in the test window, a tool schema that shipped after the canary closed, a prompt that happens to sit exactly on the candidate's weak spot: none of those get caught, because the thing that would have caught them stopped running weeks ago. Routing without a verifier is dead reckoning for exactly this reason: you're trusting that the map still matches the territory.
Shadow, canary, and per-request verification, side by side
| Cost overhead | Runs | Catches | After promotion | |
|---|---|---|---|---|
| Shadow (full mirror) | ~2x inference spend | 1-2 weeks, pre-launch | Distribution-level regressions, before any user sees them | Nothing. The mirror stops. |
| Canary (staged %) | Marginal, scoped to the slice | 1-2 weeks, ramping 1% to 100% | Real-user regressions the offline rubric missed | Nothing. The rollback logic is torn down at 100%. |
| Mirror sampling | 10-25% of full duplication | Ongoing, if someone keeps it on | Same as shadow, lower confidence | Only if it's still running, which it usually isn't |
| Per-request verification | Built into the cascade, no separate pipeline | Every request, permanently | Any single response that fails a calibrated bar, not just the aggregate | Nothing changes. It's still checking. |
What continuous verification looks like
The shape of it isn't complicated. Instead of gating a model swap once, gate every response:
def serve(prompt, cheap_model, strong_model, verifier, accept_threshold):
draft = cheap_model.generate(prompt)
score = verifier.score(prompt, draft) # calibrated, not a raw judge call
if score >= accept_threshold:
return draft, "cheap"
return strong_model.generate(prompt), "escalated"The difference from a canary gate is where the check sits. A canary asks "did the candidate hold up across last week's sample." This asks "did the candidate hold up on this exact prompt, right now," and it asks it forever, not just during a launch window. A calibrated verifier trained for the job, not a general-purpose LLM-as-judge call bolted on for the rollout, is what makes checking every request instead of a sample cheap enough to do permanently. Judging every response with a second frontier-model call would just move the cost problem instead of solving it, which is its own tax worth measuring.
Where Nadir fits
Nadir starts with the shadow-testing mechanism described above: return a decision receipt without changing the model that serves traffic, then attach measured usage and outcomes before enforcement. Complete non-streaming proxy responses can optionally enter a reference-free verifier cascade; streaming responses bypass it. AUROC 0.961 belongs to a separate reference-assisted RouterBench research experiment and is not deployed accuracy.
The response metadata make the decision auditable per request: model shows which model answered, nadir_metadata.complexity_analysis records the routing decision, and a configured, priced benchmark adds nadir_metadata.benchmark_comparison.savings_usd. Verifier outcomes appear in the cascade metadata only when that optional complete-response path runs.
What to ship this week
- If you're mid-rollout on a model swap, keep the mirror running past 100%. Even a 5% sample after full promotion catches drift that a one-time canary can't, for a fraction of the cost of full shadow mode.
- Automate the four rollback triggers, don't watch a dashboard. Guardrail trip rate, rubric regression, p99 latency, and candidate-only error clusters, wired to auto-rollback, not a Slack alert someone reads on Monday.
- Ask what happens to your quality check the day after 100%. If the honest answer is "nothing," that's the gap. See what a verifier-gated cascade catches that a benchmark-only comparison misses.
- Don't confuse a good shadow-test score with a permanent guarantee. It's a statement about last week's traffic distribution, not next month's. Start free and let the check run on every request instead of a sampled window.
Sources: [TianPan, "Releasing AI Features Without Breaking Production: Shadow Mode, Canary Deployments, and A/B Testing for LLMs," April 2026](https://tianpan.co/blog/2026-04-09-llm-gradual-rollout-shadow-canary-ab-testing). [Future AGI, "LLM Eval with Shadow Traffic and Canary Deployment in 2026"](https://futureagi.com/blog/llm-eval-shadow-traffic-canary-2026/).