The Retry Tax

Abstract.

Model routing is pitched as a straightforward trade: classify a prompt, send it to the cheapest model that can handle it, keep most of the savings. The pitch assumes the router knows when it guessed wrong. Most production routers don't check. They score the prompt once, before generation, and count the savings as booked the moment the request goes out. When the cheap model's answer is wrong, someone has to notice and retry, and if that retry loop is blind (same tier, same prompt, no new signal) every wasted attempt is billed in full before the task ever reaches a model that could actually solve it. This post builds a break-even model for that failure mode: given a price ratio between a cheap and a premium model, and a number of blind retries a naive router attempts before escalating, at what task failure rate does routing stop saving money at all? The formula is short: p_be = (r − 1) / (r + k − 1). At realistic retry counts it collapses fast. The fix already exists in production cascade systems, escalate on the first sign of failure instead of the fourth, and it is cheaper than what most naive routers do today.

All data and charts in this post are illustrative, modeled from a break-even framework, not measured production traces. Not derived from proprietary customer data. Sources cited throughout.

The research question.

Small-model delegation, sending simple tasks to a cheap model and reserving a premium model for hard ones, is the most common lever teams reach for first when an LLM bill gets too big. The pitch is simple: if a $0.02 model can do 70% of your traffic as well as a $0.10 model, route the 70% and keep the difference.

That pitch has a silent assumption baked in: that a wrong answer from the cheap model is free to detect and cheap to recover from. In practice, most routers are pre-generation classifiers. They look at the prompt, predict a difficulty tier, and send the request. We've written before about why that shape of router is a guess, not a measurement: it commits to a model before anyone has seen an answer. When the guess is wrong, the system needs a second mechanism to catch it, and what that mechanism does next determines whether the routing decision actually saved money or just moved the cost somewhere less visible.

Two very different things can happen next. A verifier can check the cheap model's answer once and escalate immediately on failure. Or, more common in systems built without a dedicated verification step, the caller (a retry wrapper, an orchestration framework's default error handling, or a human clicking "regenerate") can retry the same cheap tier one or more times before giving up and calling the premium model. The second pattern is what this post calls the retry tax: blind retries that add cost and latency without adding information.

The research question: given a price ratio between a cheap and a premium model, and a number of blind retries a router attempts before escalating, at what task escalation rate does small-model delegation stop saving money at all, and how does that break-even point move as blind retries are replaced with a single verification step?

Methodology.

This is an analytical framework, not a measured benchmark. Every number below is illustrative and stated as an assumption, not a claim about any specific production system. The framework generalizes; the constants don't need to match your workload to be useful, only the shape of the tradeoff does.

Core assumptions used throughout:

The central relationship: a naive router that retries blindly on the cheap tier pays for every discarded attempt, plus the eventual escalation. A verified cascade pays a small, fixed verification cost on every task and escalates directly on the first rejection. The cost of an escalated task under each design is:

• Blind retry (k retries): k × cheap_cost + premium_cost
• Verified (k = 0): verifier_cost + premium_cost

Everything downstream, the charts, the tables, the break-even curve, follows from sweeping these two designs across price ratios and escalation rates.

Where the tokens actually go.

A task that resolves cleanly on the first pass and a task that gets escalated after blind retries don't just differ in which model answers, they differ in how many times the entire pipeline runs. Reasoning, query generation, retrieved context, and result reading all repeat on every blind attempt, not just the final one. In this illustrative model, a clean pass bills 2,400 tokens; an escalated task that burns 4 blind retries before falling back to the premium model bills 12,000, a 5x multiplier that has nothing to do with the premium model's own overhead and everything to do with the retries. Retrieval and context overhead compound the same way inside a single step; the retry tax compounds it again across attempts.

The retry tax on the naive router.

Blind routing captures real savings, 32% is not nothing, but it leaves a specific amount of money on the table: the cost of 4 wasted cheap-model attempts on every one of the 30% of tasks that needed escalation anyway. Compression helps because it shrinks every attempt, wasted or not, which is why it recovers another 13.6 points without touching the retry logic at all. The full recovery, another 12.9 points on top of that, comes from removing the wasted retries entirely: a verifier that checks the first answer and escalates on rejection, instead of hoping the second, third, or fourth blind attempt does better. Stacking routing, compression, and caching together doesn't add linearly, and the same is true here: compression and verification attack different parts of the same bill, and the order they're applied in changes how much each one has left to save.

The break-even point collapses fast.

This is the finding that matters most. Blind retries don't just tax the cost of routing, they change the escalation rate at which routing is worth doing at all.

The relationship has a closed form. Let r be the price ratio between the premium and cheap model, and k be the number of blind retries a router attempts on the cheap tier before falling back. The escalation rate at which routing savings hit exactly zero is:

p_be = (r − 1) / (r + k − 1)

At r = 5 and k = 0 (a verifier that escalates on the first rejection), p_be = 1.0: routing can never cost more than staying on the premium model, no matter how bad the underlying task mix is, because the only overhead on a wrong answer is the verifier's own small cost. At k = 2, p_be drops to 0.667. At k = 4, a fairly ordinary retry-then-give-up pattern in systems without a confidence check, p_be drops to 0.5: once half your traffic needs the premium model anyway, blind retrying has already erased the entire benefit of routing. At k = 8, the kind of resampling loop that shows up in naive multi-agent frameworks trying the same subagent repeatedly before an orchestrator gives up on it, p_be falls to 0.333.

This is the mechanism behind FrugalGPT's cascade architecture and the reason verifier-gated cascades outperform pre-generation classifiers on RouterBench: the cascade's advantage isn't just that it's accurate, it's that its failure mode is bounded. A verified cascade with k = 0 degrades gracefully toward the cost of the premium-only baseline as task difficulty increases. A blind-retry router degrades past it, and does so faster the more retries it allows before giving up.

The retry-storm pattern itself isn't new to LLM systems. Distributed systems engineering has been here before: retrying a failing call against the same backend, without backoff or a circuit breaker, is a well-documented way to turn a small failure rate into an outage. AWS's Builders' Library covers the same shape of problem in the context of network calls, and Google's SRE book devotes a full chapter to how naive retries cause cascading failures. An LLM router that blindly retries a cheap model is the same failure mode with a token bill attached instead of a downtime page.

Latency: the tax hides in the tail.

The median is a misleading number here. The blind retry loop has the best P50 of the four strategies, because 70% of tasks resolve on the first cheap-model attempt and never touch the retry path. Its P95, the number a real user or downstream agent actually experiences on a bad day, is the worst by a wide margin: four sequential cheap-model calls before an escalation adds up. A verified cascade trades a small latency cost on every request (the verifier check, which runs even on tasks that would have succeeded anyway) for a much smaller latency cost on the failure path, since there's no retry loop to run through. Firing the cheap and premium model in parallel and using whichever result the verifier accepts first cuts the tail further, at the cost of paying for a premium-model call on every task, whether it's needed or not, unless that call can be cancelled the moment the verifier clears the cheap answer.

Cost versus quality: small-model-first is dominated.

The uncomfortable point in this chart isn't that blind small-model-first delegation sacrifices quality for cost, that tradeoff would at least be a choice. It's that hybrid routing costs less than blind delegation and holds quality closer to the premium baseline. Blind small-model-first is not on the cost-quality frontier at all: it is strictly worse than verified routing on both axes simultaneously, because the money it fails to save on retries buys it nothing in return. The only reason to end up there is not measuring escalation cost separately from routing savings in the first place.

What this means for engineering teams.

A few tradeoffs worth stating plainly, because none of the levers here are free:

• Reducing cost too aggressively still hurts quality, just not always where you'd expect. Aggressive compression alone lands at 85% quality in this model, worse than a verified cascade that costs more but catches failures before they ship. Cost reduction and quality preservation are only aligned when the technique targets overhead, not decision-critical content.
• Small models are not always cheaper once retries are counted. A 5x price ratio sounds like a strong case for routing. It stops being a strong case the moment a naive retry loop is allowed to run 4 or 8 times before giving up, because each blind retry pays the cheap-model price for zero new information.
• Caching only pays off when the workload repeats. The 15% caching discount used in this model assumes a meaningful hit rate on repeated prefixes; a workload with unique, non-repeating context per task won't see it, and shouldn't budget for it.
• Parallel speculative execution buys latency at the cost of extra spend, unless the losing call (the premium model's answer, when the cheap model's passed verification) can be reliably cancelled before it's billed. If it can't, speculative parallelism is a latency win and a cost regression at the same time.
• Model routing needs monitoring, not a fixed retry count. A hardcoded "retry twice then escalate" policy is a guess about your task mix baked into code. The break-even formula above depends on your actual price ratio and your actual failure rate, both of which drift as models, prompts, and traffic change.

Implementation framework.

Step 1: Instrument escalation events separately from routing decisions. Log whether each task resolved on the cheap tier, escalated after verification, or escalated after N blind retries. Most teams that measure "router savings" are actually measuring routing decisions made, not tasks successfully resolved at the cheap tier, and the gap between those two numbers is the retry tax.

Step 2: Replace fixed retry counts with a verification step. A calibrated verifier that scores the cheap model's answer before shipping it, the same architecture behind our binary classifier's routing decisions, turns k blind retries into k = 0. This is the single highest-leverage change in the framework above: it's the difference between a break-even point that never crosses zero and one that collapses at 33-50% escalation.

Step 3: Measure your actual price ratio and escalation rate. The formula p_be = (r − 1) / (r + k − 1) is only useful with real inputs. Pull your cheap and premium model's blended cost per task, and pull your actual escalation rate from the logs in Step 1, not an assumption.

def resolve_task(task, cheap_model, premium_model, verifier, max_blind_retries=0):
    """Verifier-gated cascade: escalate on the first verifier rejection.
    Set max_blind_retries > 0 only to reproduce the naive blind-retry pattern
    for comparison; production routing should default to 0."""
    cost = 0.0
    for attempt in range(max_blind_retries + 1):
        answer = cheap_model.run(task)
        cost += cheap_model.price_per_call
        verdict = verifier.check(task, answer)
        cost += verifier.price_per_call
        if verdict.passed:
            return answer, cost, {"tier": "cheap", "attempts": attempt + 1}
    answer = premium_model.run(task)
    cost += premium_model.price_per_call
    return answer, cost, {"tier": "premium", "attempts": max_blind_retries + 1}

Step 4: Route simple, classifiable steps to cheaper models, but don't stop there. Routing without verification is a guess about which tier a prompt needs, made before generation. It should still be the first filter; it just can't be the only safety net.

Step 5: Compress after routing, not instead of it. Compression reduces the cost of every attempt, cheap or premium, wasted or not. It narrows the retry tax; it doesn't remove it. Apply it on top of a verified cascade, not as a substitute for one.

Step 6: Cache what actually repeats. System prompts, tool schemas, and other static prefixes are worth caching. Don't budget caching savings against dynamic, per-task context that won't hit a second time.

Step 7: Monitor cost, latency, and quality together, per tier. A router that looks like it's saving money on a cost-per-request dashboard can be quietly retrying its way through the savings on the tasks that fail. Track completed-task cost (including retries and escalations), P95 latency, and quality by tier, not a single blended average.

What this doesn't solve.

This is a two-tier, single-verifier model. Production systems often have more tiers, imperfect verifiers with their own false-positive and false-negative rates, and retry policies that vary by task type rather than a single global k. A verifier that is wrong in the "pass" direction ships bad answers without ever triggering escalation, which this framework doesn't price in, that failure mode needs its own calibration measurement, not a break-even formula. The price ratio r is also treated as fixed here; in practice it shifts with model pricing changes and with how much context each tier actually consumes, which is its own moving target.

Conclusion.

The instinct to route simple tasks to cheap models is correct. The instinct to assume that decision is free to get wrong is not. A router that can't tell the difference between "this needs a retry" and "this needs to escalate" defaults to retrying, and every blind retry is billed at full price for producing nothing. The future of small-model delegation isn't just picking the right price ratio, it's pairing that decision with a cheap, fast way to know when it was wrong.

This is the problem verifier-gated cascades are built to solve: instead of hoping the cheap model gets it right, or retrying blind and hoping the next attempt does, check the answer once and escalate immediately when the check fails. Nadir runs that check on every routed request, so escalation happens on the first sign of trouble instead of the fourth, and the savings dashboard shows the real delta, retries and all, against always using the premium model.

Teams that separate "routing decisions made" from "tasks resolved at the cheap tier" consistently find the same gap: some of what looks like router savings is actually a retry loop nobody priced. Closing that gap doesn't require a bigger model or a smarter classifier. It requires knowing, cheaply and quickly, when the first answer was wrong.

Data and charts in this post are illustrative, derived from the break-even framework described above, not measured production traces. Sources: [Chen, Zaharia, Zou, "FrugalGPT: How to Use Large Language Models While Reducing Cost and Improving Performance," arXiv:2305.05176, Stanford, TMLR 2024](https://arxiv.org/abs/2305.05176). [Brooker, "Timeouts, retries, and backoff with jitter," AWS Builders' Library](https://aws.amazon.com/builders-library/timeouts-retries-and-backoff-with-jitter/). [Google SRE Book, "Addressing Cascading Failures"](https://sre.google/sre-book/addressing-cascading-failures/).