Your Router Is Fighting Your Prompt Cache

The discount that belongs to one model

Prompt caching is the closest thing to free money in the hosted LLM stack, and this blog has made that case before: mark a stable prefix, pay to write it once, and every request that repeats it within the TTL bills that prefix at a fraction of list price. On Anthropic a cache read costs 10% of fresh input. On OpenAI it is about 50%, automatic, with no write fee. On Gemini, about 25%. Source: Anthropic, "Prompt caching"; Source: OpenAI, "Prompt caching".

The part that gets skipped: that discount is keyed to a model. The cache Anthropic keeps for your 50,000-token agent scaffold lives under claude-sonnet-4-6 specifically. Route the next turn of that same session to claude-haiku-4-5 and, as far as the cache is concerned, you are a stranger. The prefix processes from scratch at Haiku's rates, plus a 1.25x cache-write premium if you keep caching, which you should. And the 90% read discount Sonnet had already earned is simply forfeited.

A model router makes exactly that move on purpose, on every request. Per-request routing, this blog's whole thesis, asks "which is the cheapest adequate model for this prompt?" and sends the easy turn to the cheap model. Prompt caching quietly makes that question wrong for multi-turn sessions. The cheapest model for this request and the cheapest path for this session stop being the same thing once a large prefix is warm somewhere, because cost is no longer a property of the request. It is a property of the session's cache state.

The arithmetic of a switch

Put real prices on it. Claude Sonnet bills $3 per million input tokens and $15 per million output. Haiku bills $1 and $5, three times cheaper on paper. Cache reads are 10% of input price, cache writes 1.25x for the default 5-minute TTL. Source: Anthropic, "Pricing".

Now take a coding-agent session with a 50,000-token cached prefix, system prompt, tool schemas, and accumulated context, warm on Sonnet. The next turn adds 2,000 fresh input tokens and produces about 500 output tokens.

Stay on warm Sonnet. The prefix bills at the read rate, the new tokens at list:

50,000 x $0.30/M + 2,000 x $3/M + 500 x $15/M = $0.0285

Switch to cold Haiku. The prefix re-bills at Haiku's write rate, then Haiku's cheaper rates apply to the turn:

50,000 x $1.25/M + 2,000 x $1/M + 500 x $5/M = $0.0670

The switch costs 2.4x more than staying, for a model that costs a third as much per token. Per prefix token, the comparison is a $0.30/M cache read against a $1.25/M cache write, a 4.2x spread before Haiku's cheaper fresh tokens claw anything back. At one thousand such sessions a day, cache-blind downrouting on that single turn is the difference between $28.50 and $67.00, roughly $14,000 a year, spent by the cost-optimization layer.

The cache-write tax post covered writes as a standalone line item. A switching router pays that tax repeatedly, and pays it precisely when it believes it is saving money.

The variable that decides it: turns remaining

Switching is not always wrong. It pays a one-time cost, the prefix re-write, and earns a recurring per-turn saving, the cheaper model's rates on everything after the prefix. Whether it nets out depends on how long the session keeps going. Over N remaining turns, with prefix P and a per-turn load of T_new fresh input and T_out output tokens:

stay   = N x (P x read_warm + T_new x in_warm + T_out x out_warm)
switch = P x write_new + (N - 1) x P x read_new
         + N x (T_new x in_new + T_out x out_new)

Run the 50,000-token example across horizons:

The break-even in this example sits near three turns. A support bot wrapping up a conversation should stay warm. An agent loop settling in for forty tool calls should eat the re-write immediately, because Haiku warms its own cache on turn one and every turn after that bills the prefix at Haiku's own 10% read rate.

The intuition compresses to one line: the bigger the cached prefix is relative to each turn's fresh work, the longer you should stay on the model that holds it. Agent scaffolds are exactly the workloads with huge stable prefixes and small per-turn deltas, which is why this bites agent traffic hardest, the same traffic where structural overhead already dominates the token budget.

Quality still rules, cost only breaks ties

One guardrail before any of this ships: cache math must never route a request below the quality it needs. If the session is warm on Haiku and the next turn is genuinely Opus-tier work, the router should escalate and eat the cache loss. The stay-or-switch comparison is legitimate only between candidates already admitted by policy. Nadir's older 60% / 98% RouterBench figure is a reference-assisted research ceiling, not evidence that the deployed cache-aware path universally clears that constraint.

There is a second, quieter half of this contract: whatever optimizes your context must not invalidate the prefix either. Compression that rewrites an earlier message re-bills the whole cached prefix at write rates, which can cost more than the tokens it removed. Nadir's /v1/optimize holds a prefix-stability contract for exactly this reason: re-optimizing a conversation that has grown by a turn reproduces the previously optimized messages byte-identically, so the cache keyed on the optimized prefix keeps hitting.

Making the router cache-aware

The fix is to hand the router the one fact it is missing, what is warm and how big it is. Nadir's decision endpoint takes it as an optional field:

curl https://api.getnadir.com/v1/recommend \
  -H "X-API-Key: $NADIR_API_KEY" \
  -d '{
    "prompt": "Apply the fix we discussed to the parser.",
    "cache_state": {
      "model": "claude-sonnet-4-6",
      "cached_tokens": 50000,
      "ttl": "5m",
      "expected_remaining_turns": 2
    }
  }'

The response carries the fresh pick, the cache-adjusted decision, and both projected costs, so the override is auditable rather than magical:

{
  "recommended_model": "claude-sonnet-4-6",
  "selection_method": "cache_aware_stay",
  "cache_advice": {
    "decision": "stay_warm",
    "router_model": "claude-haiku-4-5",
    "stay_cost_usd": 0.057,
    "switch_cost_usd": 0.0765,
    "switch_penalty_usd": 0.0195,
    "horizon_turns": 2
  }
}

If the cache has expired, the quality bar rules the warm model out, or the horizon is long enough that switching pays, the fresh pick stands and cache_advice says why. The same economics surface in the dashboard: the Analytics cost view now tracks cache reads against cache writes per model, with the one warning worth automating, writes exceeding reads, which means you are paying the 1.25x premium for prefixes that expire before anything reads them back.

If you run your own router

The checklist is the same whether or not Nadir is in the loop:

• Log cache reads and writes per request. Both Anthropic and OpenAI return the counts in the usage block. If your gateway drops them, you cannot see any of this.
• Make the switch decision session-aware. A per-request router needs the session's warm model and prefix size as an input, or it will downroute into the write premium forever.
• Prefer switching at session boundaries. A new session has no warm prefix, so the cheap model starts clean and earns its own discount from turn one.
• Watch the write-to-read ratio. Sustained writes above reads means paying premiums for caches nothing reuses, the signature of either cache-blind switching or a TTL shorter than your think time.
• Keep prefixes byte-stable. Anything that rewrites earlier messages, compression, re-ordered tools, timestamps in system prompts, silently turns reads back into writes.

Routing and caching are the two biggest levers on an LLM bill, and most stacks operate them as if they were independent. They are not: every routing decision is also a cache decision. Price them together and the router stops fighting the cache and starts using it, which is the difference between a router that optimizes requests and one that optimizes your bill. For the rest of that playbook, start with the complete guide to cutting LLM API costs.