Route Like a Bank

TL;DR.

In one week in late October, three of the largest AI buyers in the world described the same cost strategy in public. None of them said they'd buy more of the newest model. All three said they'd stop sending every request to it.

• Bank of America routes. Its routing layer, Orchestra, uses a small classifier model to send each query to the cheapest model that can handle it. The bank still plans to double its AI budget in 2027.
• Capital One distills. It distilled fine-tuned open-weight models into smaller ones for Chat Concierge, its car-buying assistant, and reports 5x lower latency and 50% more dealer engagement.
• SAP swaps. It runs more than 70 models, pulled some of the most expensive ones out of products like its internal consulting assistant, and expects more than €2 billion in efficiency gains over two years.
• Everyone else is cutting. In Cloudera's State of Enterprise AI 2026 report, 89% of organizations scaled back or redesigned AI plans this year because of token costs. Only 23% say AI produces more value than it costs.
• In our model the three playbooks aren't equal. On an assistant serving 1M requests a month, swapping to Sonnet 5.5 saves 50%. Routing per request saves 70%. Distilling the easy tier onto your own GPUs costs more than routing until about 6.2M requests a month, because Haiku 5.5 is already $0.10 per million input tokens.
• Below is orchestra.py, 83 lines: a policy filter, a cheap label, a check that escalates one tier, a log line per request, and price(), which reproduces every number here.

Company facts come from Banking Dive and CIO Dive coverage of the Evident AI Symposium (October 22, 2026), Reuters on SAP (October 28, 2026), and ITPro on Cloudera's report (October 28, 2026), all retrieved November 2, 2026. Prices are Anthropic list prices on the same date. The traffic model (1M requests, 3,000 tokens in, 400 out, a 60/30/10 split, 5% misroutes, $3,650 a month of GPUs) is ours and illustrative. No figure here comes from Nadir customer data.

Three companies, one week.

Gopalkrishnan's reasoning was the simplest version of the routing case: "there are simpler classifier models that do very easy decision things that you don't need a large language model for." He also gave the reason that has nothing to do with price: "We didn't want to be beholden to any given model."

SAP's framing matters just as much. It still expects AI spending to rise as adoption grows. The goal, as SAP described it, is healthy margins at scale, not a smaller AI bill. None of these companies is cutting AI. They're cutting the average price of a request so they can afford more requests.

Why now.

The pressure is broad. Cloudera's report, covered by ITPro on October 28, found:

• 89% of organizations scaled back or redesigned AI plans this year because of token costs.
• 45% moved AI workloads because of higher or more volatile costs.
• 23% say AI generates more value than it costs.
• 27% can measure end-to-end AI cost with high accuracy, and 19% monitor it continuously.

Unit prices aren't the problem. They fell. Per-request spend climbed anyway because agents call models many times per task, and because the default model in most stacks is still the expensive one. We covered that gap in Cheap Tokens, Empty Budgets. What changed in October is that large buyers stopped treating the default model as a given.

What each playbook costs.

Take one internal assistant: 1M requests a month, 3,000 tokens in and 400 out per request. In our model, 60% are simple (lookups, FAQ answers, extraction), 30% are medium (a multi-step answer, a short analysis), and 10% are complex (judgment calls, long reasoning). Here's what each playbook costs on Anthropic's current list prices.

Swap is the easy half. It needs one eval per product and no new infrastructure. It also caps the saving. A product-wide swap can't go cheaper than the model that handles its hardest common request, and the 10% of complex requests now run on a weaker model. SAP said it only swaps where performance holds, and that's the right condition.

Routing captures the spread inside each product. Sixty percent of requests on Haiku 5.5 cost $0.0005 each, and the 10% on Opus cost $0.020. Routing pays for itself even after you count what it costs. Our $5,944 includes a Haiku 5.5 label call on every request ($301.50 a month), simple requests over-routed to Sonnet 5% of the time ($285), and medium and complex requests under-routed 5% of the time and paid for twice after the check catches them ($57.50).

Distillation is not a cost play at this volume. That's the counterintuitive result. Two always-on GPUs at an illustrative $2.50 an hour cost $3,650 a month. Routing the same 600,000 simple requests through the API costs about $585, including the 5% that land on Sonnet. Capital One's reported wins were latency and engagement, not token cost, and that matches the arithmetic.

When distilling pays.

A self-hosted model trades a per-request price for a fixed one. In our model the lines cross at about 6.2M requests a month, or 3.7M simple ones. Above that, the distilled tier is cheaper, and the gap grows: $2,200 a month at 10M requests.

The fixed cost in the chart is only the GPUs. A real distillation project also pays for training data, training runs, an eval set, and the people who keep the model current when the base model or the task changes. If your GPUs sit half idle, the effective price doubles, which we covered in The Utilization Tax.

So distill when one of these is true:

• Latency is the product. A car buyer waiting in a chat window is Capital One's case. A small model on your own hardware has no queue you don't control.
• The task is narrow and stable. One domain, one output shape, and proprietary data the frontier model never saw.
• Volume is far past break-even. Tens of millions of requests a month in a single narrow tier.
• The data can't leave. Then it isn't a cost decision at all.

Otherwise, route to a hosted small model first. It's the same architecture with no fixed cost, and you can distill the busiest tier later, once your logs show how busy it really is.

The four parts of a routing layer.

The banks didn't publish their designs, but what they did say fills in the outline.

1. Policy before price. SAP said government agencies and regulated industries increasingly exclude models by country of origin, so the allowed set differs per customer. The router has to filter to permitted providers before it picks the cheapest one. Choosing the cheapest model first and checking policy afterward is how a request ends up somewhere it isn't allowed to go. Residency works the same way, as we showed in The Residency Tax.

2. A cheap label. Gopalkrishnan described a small classifier model in front of the larger ones. Rules come first: short, single-question requests go straight to the small tier and cost nothing to label. Everything else gets one call to the cheapest allowed model that returns one word. When the label is unclear, default to the top tier. A misroute upward costs money. A misroute downward costs a wrong answer.

3. A check, then one step up. Validate what you can cheaply: JSON parses, required fields are present, the answer isn't empty, the length is in range. On a fail, retry one tier up and pay twice. In our model that retry costs $57.50 a month at a 5% miss rate. The check won't catch a fluent wrong answer. That's the limit of routing without verification, and it's why the miss rate has to be measured, not assumed.

4. A log line per request. Gopalkrishnan listed "the instrumentation, telemetry, being able to have a feedback loop" as what the bank is still building. Cloudera's 19% figure says most companies don't have it either. Log customer, model, pass or fail, latency, and dollars on every request. That log is how you tune the label threshold, find the tier worth distilling, and answer the CFO.

Tutorial: orchestra.py.

route() runs one request through all four parts. check is your validator: bool (non-empty) by default, or a JSON schema check, a regex, or a length limit. log defaults to print, so point it at your log pipeline. price() is the cost model behind every chart above.

"""orchestra.py: the bank pattern in one file. Filter, classify, route, check.

route() sends each request to the cheapest model this customer is allowed to
use that its class needs, checks the answer, and escalates one tier on a fail.
price() compares that with one frontier model everywhere and a static swap.
client is any OpenAI compatible client, for example OpenAI(base_url=..., api_key=...).
"""
import json
import sys
import time

# name, provider, $ per million input, $ per million output, cheapest first
TIERS = [("claude-haiku-5-5", "anthropic", 0.10, 0.50),
         ("claude-sonnet-5-5", "anthropic", 2.00, 10.00),
         ("claude-opus-5-5", "anthropic", 4.00, 20.00)]
CLASSES = {"simple": 0, "medium": 1, "complex": 2}
ALLOWED = {"default": {"anthropic", "openai", "google"},
           "eu-bank": {"anthropic", "mistral"}}   # origin rules live here
LABEL = """Label the request simple, medium, or complex. Reply with one word.
simple: lookup, FAQ, extraction, rewrite. medium: multi-step answer, short
analysis. complex: judgment, long reasoning, anything you are unsure about."""


def classify(client, prompt, model):
    if len(prompt) < 200 and "?" in prompt:
        return 0                                  # rules first, zero tokens
    r = client.chat.completions.create(
        model=model, max_tokens=3,
        messages=[{"role": "system", "content": LABEL},
                  {"role": "user", "content": prompt[:4000]}])
    return CLASSES.get(r.choices[0].message.content.strip().lower(), 2)


def route(client, prompt, customer="default", check=bool, log=print):
    tiers = [t for t in TIERS if t[1] in ALLOWED.get(customer, set())]
    if not tiers:
        raise PermissionError(f"no allowed model for {customer}")
    i = min(classify(client, prompt, tiers[0][0]), len(tiers) - 1)
    while True:
        model, _, p_in, p_out = tiers[i]
        t0 = time.time()
        r = client.chat.completions.create(
            model=model, messages=[{"role": "user", "content": prompt}])
        text, u = r.choices[0].message.content, r.usage
        ok = check(text)
        log(json.dumps({"customer": customer, "model": model, "ok": ok,
                        "ms": int((time.time() - t0) * 1000),
                        "usd": round((u.prompt_tokens * p_in +
                                      u.completion_tokens * p_out) / 1e6, 6)}))
        if ok or i == len(tiers) - 1:
            return text                           # top tier answers anyway
        i += 1                                    # failed check: one tier up


def cost(tier, tin=3_000, tout=400):
    _, _, p_in, p_out = TIERS[tier]
    return (tin * p_in + tout * p_out) / 1e6


def price(n=1_000_000, mix=(0.60, 0.30, 0.10), miss=0.05, gpu_month=0):
    """Monthly cost: frontier everywhere, static swap, per-request routing."""
    frontier = n * cost(2)
    swap = n * cost(1)
    label = n * (3_000 * TIERS[0][2] + 3 * TIERS[0][3]) / 1e6
    routed = 0.0
    for tier, share in enumerate(mix):
        right = cost(tier)
        if tier == 0:                             # too high: overpay, no retry
            each = (1 - miss) * right + miss * cost(1)
        else:                                     # too low: pay twice
            each = right + miss * cost(tier - 1)
        routed += n * share * each
    if gpu_month:                                 # distilled model takes simple
        routed += gpu_month - n * mix[0] * ((1 - miss) * cost(0) + miss * cost(1))
    return [round(x, 2) for x in (frontier, swap, routed + label)]


if __name__ == "__main__":
    for n in map(int, sys.argv[1:] or [100_000, 1_000_000, 10_000_000]):
        f, s, r = price(n)
        d = price(n, gpu_month=3_650)[2]
        print(f"{n:>10,} req/mo  frontier ${f:>9,.0f}  swap ${s:>9,.0f}  "
              f"route ${r:>9,.0f} ({1 - r / f:.0%} less)  route+distill ${d:>9,.0f}")

python orchestra.py prints:

   100,000 req/mo  frontier $    2,000  swap $    1,000  route $      594 (70% less)  route+distill $    4,186
 1,000,000 req/mo  frontier $   20,000  swap $   10,000  route $    5,944 (70% less)  route+distill $    9,009
10,000,000 req/mo  frontier $  200,000  swap $  100,000  route $   59,440 (70% less)  route+distill $   57,240

How to adapt it:

1. Start in shadow. Run classify() on a week of real traffic without changing which model answers. The label distribution tells you your actual mix, and your mix decides your saving. With 20% simple, 40% medium, and 40% complex, the same code saves 36% instead of 70%.
2. Put the policy in data, not code. ALLOWED belongs in the same table as the customer's contract. When a regulated customer drops a provider, routing for that customer changes the same day.
3. Make the check as strict as the task allows. Structured outputs get a schema. Free text gets length and refusal checks. Anything you can't check cheaply should be labeled complex.
4. Watch the fail rate per tier. If more than about 10% of Haiku answers fail the check, the label is too generous. At a 10% miss rate our model costs $6,286.50, still 69% under frontier, but every caught miss also adds latency.
5. Re-run price() when prices move. Gemini 4 Argon doubles on December 1, and more scheduled changes follow through January 14, as we listed in Seven Dates, One Bill. A static swap made in October can be the wrong swap by January. A router re-ranks on the next request.

When not to bother.

• Most of your traffic is hard. If the label puts most requests in the top tier, routing adds a call and saves little. Use a cheaper frontier model for the whole product and move on.
• You serve one tiny model already. If everything runs on Haiku-class pricing, the label call is a meaningful share of the bill. Use rules only, or nothing.
• Nobody owns the eval. Every playbook here, swap included, depends on an eval that says the cheaper model is good enough. Without one, you're just guessing.
• Spend is small. At 100,000 requests a month the frontier bill in our model is $2,000. Routing saves about $1,400. That's real, but it may not justify building and running your own layer.

Where Nadir fits.

orchestra.py is the shape of the pattern. Nadir is that pattern as a hosted service, so you don't have to build and maintain your own Orchestra.

The label and the route through one endpoint. Point your OpenAI client at Nadir's OpenAI compatible gateway and set model="auto". A trained classifier sends each request to the cheapest model you've allowed that clears your quality floor. POST /v1/bucket labels a prompt simple, medium, or complex without a provider call, which is a drop-in replacement for classify() if you'd rather keep your own routing code.

Policy per key. Each API key has its own model pool, allowlist, and quality floor, so a regulated customer and an internal tool can run through the same gateway with different rules. A fallback must pass the same allowlist as the primary, so an outage can't push a request onto a model the key doesn't permit. Pin a model when a workload needs exactly one.

The log line, built in. Every response reports the model that served it and what it cost, so step 4 doesn't need a separate billing export.

Measure before you switch. Observe mode serves exactly the model you asked for and records what routing would have chosen, so you can see your real mix and the saving on a week of traffic before routing anything. With your own provider keys, Nadir charges no fee on any plan, and the provider bills you directly. Savings depend on your workload, and nothing in this post forecasts yours.

Start with a free key, or estimate your savings from your current model spend.

Conclusion.

Bank of America, Capital One, and SAP aren't spending less on AI. Bank of America is doubling its budget, and SAP expects spending to keep rising. What they've stopped doing is paying frontier prices for requests that don't need a frontier model. Swapping is the quick win and caps out at the price of your hardest common request. Routing per request gets the spread inside every product. Distilling pays for latency and control first and for cost only at very high volume. If 89% of companies are redesigning their AI plans around token cost, the ones that put the routing layer in first will have the most room to grow.

Company facts: [Banking Dive, "Smaller AI models help Bank of America, Capital One manage costs," October 29, 2026](https://www.bankingdive.com/news/bank-of-america-gopalkrishnan-capital-one-natarajan-manage-ai-costs/873146/). [CIO Dive, coverage of the same Evident AI Symposium session](https://www.ciodive.com/news/bank-of-america-capital-one-manage-ai-costs/873112/). [Reuters via The Star, "SAP swaps costly AI models for cheaper alternatives, targets over €2 billion in gains," October 28, 2026](https://www.thestar.com.my/tech/tech-news/2026/10/28/sap-swaps-costly-ai-models-for-cheaper-alternatives-targets-over-2-billion-in-gains). [ITPro, "89% of enterprises have scaled back AI deployments over token costs," October 28, 2026](https://www.itpro.com/technology/artificial-intelligence/ais-price-tag-is-starting-to-shape-its-path-through-the-enterprise-89-percent-of-enterprises-have-scaled-back-ai-deployments-over-token-costs-and-theyre-still-struggling-to-track-spending). [Anthropic, Pricing](https://claude.com/pricing). All retrieved November 2, 2026. Capital One's latency and engagement figures and SAP's €2 billion figure are company statements, not audited results. Prices: Opus 5.5 $4 input and $20 output, Sonnet 5.5 $2 and $10, Haiku 5.5 $0.10 and $0.50 (its rate for prompts up to 100K tokens), per million tokens. The traffic model is ours and illustrative: 1M requests a month, 3,000 input and 400 output tokens each, 60% simple, 30% medium, 10% complex, a 5% misroute rate in each class, a Haiku 5.5 label call on every request (3,000 tokens in, 3 out), and every under-route caught by the check and retried one tier up. The distilled tier is 2 GPUs at an illustrative $2.50 an hour, always on, with training, evaluation, and staff excluded.