Cache-Blind Routing

Two requests, same prompt, wildly different bill

Send the identical prompt to the same self-hosted model fleet twice, back to back. On one routing strategy, the second request returns a first token in half a second. On another, it takes a minute and a half. The prompt did not change. The model did not change. The hardware did not change. The only variable was which of the fleet's replicas the load balancer happened to pick.

That is not a hypothetical. It is the headline result from a benchmark run on an 8-pod, 16-H100 vLLM cluster simulating 150 enterprise customers with 5 concurrent users each, with cache demand equal to 73% of total cluster capacity. Source: llm-d, "KV-Cache Wins You Can See: From Prefix Caching in vLLM to Distributed Scheduling with llm-d". At P90, random (round-robin style) scheduling took 92.551 seconds to produce a first token. Prefix-aware ("precise") scheduling took 0.542 seconds. That is a 170x gap, on hardware that never changed. Source: llm-d, "KV-Cache Wins You Can See".

Every post on this blog argues that routing is the highest-leverage lever on your LLM bill: send the easy step to a cheap model, save the frontier model for the step that needs it. All of that still holds. What gets skipped is that the router itself can be the thing quietly wrecking a second, adjacent cost lever, the KV cache, and most teams running a self-hosted fleet or a homegrown gateway have never measured whether their own routing does this.

Two different things both get called "cache" here

This blog has already covered prompt caching, the billing feature Anthropic, OpenAI, and Google expose in their hosted APIs: mark a stable prefix, pay full price once, pay a fraction of it on every repeat within the cache TTL. That is a provider-side accounting decision. You do not control where the cache lives or how it is evicted. You only control whether your requests are structured to hit it.

KV cache reuse, sometimes called prefix caching at the infrastructure layer, is the mechanism underneath that accounting decision, and it is also the thing self-hosted inference stacks manage directly. Every transformer forward pass produces key and value tensors for every token it has processed. Recomputing those tensors for a prefix you have already seen, the system prompt, the tool schemas, the first eight turns of a conversation, is pure waste: identical math, identical result, paid for twice. vLLM's PagedAttention and SGLang's RadixAttention both exist to store those tensors and serve them back out on a matching prefix instead of recomputing them. Source: BentoML, "Prefix caching," LLM Inference Handbook. By 2026 this is assumed infrastructure, not a nice-to-have: production stacks serving Llama 4, Qwen 4, or DeepSeek V4 routinely tune cache block size and prefix hit rate per route. Source: DigitalApplied, "KV Cache Optimization for LLMs 2026: Engineering Guide".

The gap between the two: a hosted API's prompt cache lives inside one provider's black box. A self-hosted KV cache lives on a specific GPU, in a specific replica's memory, and it evaporates the moment a request that could have reused it gets sent somewhere else. That "somewhere else" decision is made by your router or load balancer, on every single request, usually without anyone having told it the cache exists.

Round-robin was built for stateless web servers. Inference is not stateless.

Round-robin and least-connections load balancing both assume every replica is interchangeable and holds no state worth preserving. That assumption is correct for a stateless API server and wrong for an LLM replica holding a warm KV cache. Source: TrueFoundry, "KV Cache Routing: Why Standard Load Balancers Break Prefix Caching and How to Fix It". Spread eight replicas behind round-robin and a repeat request carrying the same 6,000-token system prompt has, purely by the arithmetic of the distribution, a 1-in-8 chance of landing back on the one replica that already has it cached. The other seven have to recompute the entire prefix from scratch, at full compute cost, on every one of those misses.

This is exactly the same shape of problem this blog has already measured inside a single agent loop, where 82% of an agent's token budget goes to structural overhead before the model writes a word of the answer, and where multi-agent orchestration multiplies that overhead 15x across every hop. Cache-blind routing is the fleet-level version: instead of one agent re-paying for its own stable prefix, a whole cluster re-pays for a prefix that is already sitting in memory two racks over.

Cache-aware routing, measured

Vendors publishing benchmarks on this problem converge on the same shape of result, at different scales:

Sources: llm-d, "KV-Cache Wins You Can See"; TrueFoundry, "KV Cache Routing". These are illustrative benchmark figures from the cited studies' own hardware and workload mixes, not universal constants. Your gap depends on your replica count, your prefix length, and how much of your traffic actually repeats a prefix. That last variable turns out to matter more than any routing algorithm.

Cache-aware routing is not free, and it is not universal

The one number in that table that should make you pause before wiring prefix-affinity into everything: hit rate is workload-dependent, and the range is enormous. SwiftCache's measurements across production-style workloads found prefix cache hit rates of 90.7% for multi-turn conversation and 81.8% for question-answering and RAG, against 6.3% for summarization and 0.1% for code completion. Source: "SwiftCache: Efficient LLM Serving for Multi-turn Conversations with Heterogeneous KV Cache Sharing," arXiv:2606.16135. A chat agent or a RAG pipeline has enormous, repeatable structure to exploit: the same system prompt, the same tool schemas, the same retrieved document across a session. A summarization job or a one-shot code completion often carries a genuinely distinct context on every call. Routing for cache affinity on that traffic buys you close to nothing and can cost you load balance.

That tradeoff is real, not theoretical. Pin every request with a shared prefix to the same replica and you risk starving that replica while its seven siblings sit idle, the classic hot-spot failure mode of any sticky-routing scheme. The fix researchers have converged on is to score, not pin: weigh prefix similarity against real-time GPU memory pressure and predicted time-to-first-token, and fall back to standard load balancing the moment queue depth on the "right" replica gets too deep. Source: "CacheRoute: KV-Cache-Aware Routing for LLM Backend Services on Kubernetes". Cache affinity is one signal into the routing decision. It should never be the only one.

The fix, in the projects already shipping it

None of this requires custom infrastructure at this point. Three approaches are already in production tooling:

• Consistent hashing on a routing key. Hash the session ID, user ID, or a fixed-length prefix of the prompt, and route matching hashes to the same replica. This is the simplest version and captures most of the available gain with no custom scheduler. Source: Ray Docs, "Prefix-aware routing".
• Radix-tree or hash-based prefix matching. SGLang's RadixAttention and vLLM's own prefix cache track the longest common prefix between an incoming request and everything currently cached, and a router built on top of that (Ray Serve's PrefixCacheAffinityRouter, the vLLM Router project) scores each replica by how much of the request it already holds. Source: BentoML, "Prefix caching"; Source: vLLM Blog, "vLLM Router: A High-Performance and Prefill/Decode Aware Load Balancer for Large-scale Serving".
• Joint scoring across cache, load, and latency. CacheRoute and the llm-d scheduler both combine prefix similarity with live GPU memory utilization and a predicted TTFT, so the router degrades gracefully to load balancing instead of overloading the one replica that happens to hold the cache. Source: "CacheRoute".

The shape of the decision function is short enough to sketch:

# Score each replica: cache affinity, load, and predicted latency together
def score_replica(replica, request_prefix_hash):
    cache_overlap = replica.kv_cache.longest_match(request_prefix_hash)
    load_penalty = replica.queue_depth / replica.max_queue_depth
    predicted_ttft = estimate_ttft(replica, cache_overlap)

    # High overlap wins, but a saturated replica loses even with a warm cache
    return cache_overlap * (1 - load_penalty) - predicted_ttft

def route(request, replicas):
    return max(replicas, key=lambda r: score_replica(r, request.prefix_hash))

That is the whole idea: cache affinity is a term in the scoring function, not an override of it. A router that pins on cache alone and ignores load will eventually create its own denial-of-wallet incident, the same failure mode this blog covered from the security side, just caused by a well-intentioned optimization instead of an attacker.

The same tax hits you even if you have never run vLLM

If your traffic goes entirely to hosted APIs, you are not managing a KV cache directly, but the underlying mechanism, and the underlying failure mode, are the same. Anthropic's and OpenAI's prompt caching only pays off if requests carrying a shared prefix keep landing in a way the provider can match against a live cache entry within its TTL. A router that fans a single conversation across multiple API keys for rate-limit headroom, or that swaps models mid-session to chase a marginally cheaper rate, or that load-balances across regions inconsistently, can break that prefix match just as thoroughly as round-robin breaks a self-hosted KV cache. The bill just shows up as a flat "no cache hit" on every request instead of a slow first token, which makes it easier to miss.

The instrumentation this blog has recommended for system prompt bloat and prompt caching applies directly here: log cache hit status per request, not just cost per request, and treat a falling hit rate as a routing regression to investigate, not a fact of life.

What to ship this week

• Measure your current hit rate before touching your router. If you are self-hosting, most inference stacks expose this as a metric already. If you do not know your number, you cannot know whether a change helped.
• Route on a stable key, not randomly. Session ID, user ID, or a hash of the fixed prefix is enough to move off pure round-robin without adopting a new scheduler.
• Score cache affinity against load, never pin blind. A sticky router that ignores queue depth trades a latency problem for an availability problem.
• Segment by workload before you optimize. Chat and RAG traffic can carry a high hit rate that is worth routing around. Summarization and one-shot code completion mostly cannot, per SwiftCache's own numbers, so do not spend engineering effort chasing cache affinity there.
• If you are on hosted APIs, watch cache hit status as a routing health signal, the same way you would watch error rate. A drop usually means something upstream started fragmenting sessions across models, keys, or regions.

Conclusion

The industry spent 2025 and the first half of 2026 proving that model routing cuts LLM cost, and this blog has covered that case from a dozen angles. What the KV cache benchmarks add is a reminder that the router is not just a cost decision, it is also an infrastructure decision, and a router that only optimizes "which model" while ignoring "which replica" can hand back most of what routing was supposed to save. A 170x swing in time-to-first-token from scheduling strategy alone, on identical hardware, is not a rounding error. It is the same category of waste this blog has spent a year documenting in system prompts, tool schemas, and retrieved context, just moved one layer down the stack, into the load balancer nobody audits because it is "just infrastructure."

See where your own requests land. Nadir already reports cache hit status per request in the response metadata and the savings dashboard, and NadirClaw's custom endpoint support routes to vLLM, LocalAI, or any OpenAI-compatible backend without scattering a session across replicas by default. If you are running your own fleet, that is the first number worth pulling before you touch a single routing rule.

Related reading

• Prompt caching is the closest thing to free money in LLM pricing
• Compression saves you once. Routing plus compression saves you twice.
• Your AI research agent spends 82% of its token budget before writing a single word
• Multi-agent AI costs 15x more, and almost nobody routes it
• vLLM Semantic Router cuts tokens 48%. It also needs a platform team.

Data in the charts is drawn from the cited benchmarks and papers, not from proprietary production traces, and is labeled illustrative where the source is a single vendor's test environment. Sources: [llm-d, "KV-Cache Wins You Can See: From Prefix Caching in vLLM to Distributed Scheduling with llm-d"](https://llm-d.ai/blog/kvcache-wins-you-can-see). [TrueFoundry, "KV Cache Routing: Why Standard Load Balancers Break Prefix Caching and How to Fix It"](https://www.truefoundry.com/blog/kv-cache-routing-why-standard-load-balancers-break-prefix-caching-and-how-to-fix-it). [BentoML, "Prefix caching," LLM Inference Handbook](https://bentoml.com/llm/inference-optimization/prefix-caching). [DigitalApplied, "KV Cache Optimization for LLMs 2026: Engineering Guide"](https://www.digitalapplied.com/blog/kv-cache-optimization-techniques-2026-engineering-guide). [Ray Docs, "Prefix-aware routing"](https://docs.ray.io/en/latest/serve/llm/user-guides/prefix-aware-routing.html). [vLLM Blog, "vLLM Router: A High-Performance and Prefill/Decode Aware Load Balancer for Large-scale Serving"](https://blog.vllm.ai/2025/12/13/vllm-router-release.html). ["CacheRoute: KV-Cache-Aware Routing for LLM Backend Services on Kubernetes"](https://www.researchgate.net/publication/406925855_CacheRoute_KV-Cache-Aware_Routing_for_LLM_Backend_Services_on_Kubernetes). ["SwiftCache: Efficient LLM Serving for Multi-turn Conversations with Heterogeneous KV Cache Sharing," arXiv:2606.16135](https://arxiv.org/abs/2606.16135). ["Not All Tokens Are Worth Caching: Learning Semantic-Aware Eviction for LLM Prefix Caches," arXiv:2605.18825](https://arxiv.org/abs/2605.18825).