How to Reduce LLM API Costs by 60% Without Losing Quality

The bill that arrives three times the forecast

The most common way teams first take LLM costs seriously is the same story every time. The API invoice lands at two or three times what the model was supposed to cost, finance asks which feature spent it, and nobody can answer, because the tokens were never attributed to anything. There is no single setting that fixes this. There is a stack of six levers, and the important thing about them is that they multiply. Cut input tokens 30% and route half your traffic to a model a fifth of the price, and you have not saved 30%, you have saved much more, because the discounts compound on the same requests.

This is the complete guide to the six levers, ordered the way you should actually apply them: measurement first, then the changes ranked by return on effort. For each one you get the realistic savings range and a link to the deep dive if you want the mechanics.

Lever zero: you cannot cut what you cannot see

Before any optimization, you need per-feature cost attribution. Not a monthly total, a breakdown: cost per endpoint, per user action, per agent run. The reason is that LLM spend is almost never uniform. A single feature, a single retry loop, a single oversized system prompt usually accounts for the majority of a surprising bill, and you cannot find it from an aggregate number.

The FinOps discipline caught up to this in 2026: 98% of FinOps teams now manage AI spend, and most still cannot see where the tokens go. The fix is boring and non-negotiable. Log tokens in and out, model, and cost on every call, tagged with the feature that made it. Datadog's own telemetry found 69% of tokens were going into system prompts that most teams never inspected. You will not believe where the money goes until you measure it, and you will not measure it by accident.

Once you can see the bill, apply the levers below in order.

Lever one: route each request to the cheapest capable model

This is the largest single lever, and the one most teams skip, because the default behavior of every SDK is to send every request to whatever model you hard-coded, usually the best and most expensive one. But most requests are not hard. Meta's RouteLLM work found 74% of GPT-4-class calls did not need a GPT-4-class model to get the same answer. You are paying frontier prices for a workload that is mostly easy.

LLM routing is a decision layer that reads each request, estimates its difficulty, and sends it to the cheapest model that can still answer it well. The critical detail is verification. A router that guesses the model before seeing an answer is dead reckoning, and it ships bad responses when it guesses wrong. A verified router lets the cheap model answer, scores that answer with a calibrated verifier, and escalates only on a reject.

That distinction is the difference between routing that measures quality and routing that assumes it. A reference-assisted RouterBench research experiment projected 60% lower cost at about 98% retained quality, but the verifier had the expensive reference answer unavailable in production. Treat it as a research ceiling. Customer decisions should begin with a declared baseline, shadow traffic, and outcomes.

Estimated saving: 40 to 60% on a mixed workload.

Lever two: stop resending the same tokens uncached

Agents and chatbots resend the same large prefix on every turn: the system prompt, the tool schemas, the retrieved documents, the conversation so far. Provider prompt caching lets you pay a steep discount on that repeated prefix instead of full price every time. OpenAI and Anthropic both discount cached input tokens sharply, and for a workload with a big stable prefix that is close to free money left on the table.

Two adjacent techniques stack on top. Semantic caching returns a stored answer when a new request is similar enough to a past one, skipping the model call entirely. And on self-hosted fleets, routing across the KV cache instead of blind round-robin avoids recomputing context that already lives on a replica, a gap that produced a measured 170x swing in time-to-first-token on identical hardware.

Estimated saving: 30 to 50% of input cost on cache-heavy workloads.

Lever three: cut the input tokens you actually send

Caching discounts the tokens you resend. This lever removes tokens you never needed to send at all. The usual offenders:

• System prompt bloat. Prompts accrete instructions nobody removes. Auditing and trimming the system prompt is often the highest-return hour of work available, because that prefix is on every single request.
• Tool and function schemas. MCP tool schemas and function-calling definitions are pure overhead sent on every call. Prune the tools a given request can actually use.
• Over-retrieval in RAG. Stuffing twenty chunks when three would do is paying for retrieval you do not use. And long context windows are not free just because they fit: a model degrades on a bloated context well before its advertised limit, so you pay more for a worse answer.
• Prompt compression. Tools like LLMLingua compress the prompt itself while preserving meaning, trading a little preprocessing for fewer input tokens.

Estimated saving: 15 to 40% of input cost.

Context selection: the version of this lever we measured

Of everything in that list, one operation is worth spelling out, because it is the only one that reliably removes billed tokens instead of fighting the tokenizer. Character-level tricks do not work: we tested dropping letters, gzip, and base64, and every one made text more expensive, because a mangled word usually costs more sub-word tokens than the intact one did. What works is dropping whole passages the request does not need.

That is context selection, and it runs like this per request:

1. Take the query. The latest user message is the question. It is never pruned, and neither is the system prompt.
2. Find the big blocks. Only retrieved context and tool output above roughly 800 characters are eligible. Parseable JSON is skipped so tool results stay loadable.
3. Split at boundaries. Each block splits into chunks on natural breaks. Kept chunks survive byte for byte, nothing is rewritten or summarized.
4. Score against the query. Chunks and query are embedded and scored by relevance. The encoder is the whole ballgame: a lexical word-overlap scorer saved more tokens and failed the quality gate at a 12.1% regression rate, and a semantic embedding model dropped that to 1.4%.
5. Keep the top fraction, then apply the guards. The lead and tail chunks always survive, safety and policy language is never dropped, and any chunk carrying a rare term from the question is pinned. That last guard is why the answer paragraph survives even at tight budgets.
6. Collapse the rest. Dropped runs become a single marker line, so the model can see something was omitted rather than silently missing it.

Measured on HotpotQA at a keep ratio of 0.6, with every answer graded blind by an independent judge against the known-correct answer:

On the largest contexts the answers got better while a third of the tokens left, because long noisy context degrades a model by itself and selection removes distractors along with the bill. That is the shape of this lever at its best: on exactly the bloated-retrieval traffic where the savings are largest, quality does not have to be the thing you trade.

It ships as an opt-in Nadir optimize mode, "optimize": "select", with an interactive walkthrough of the algorithm on the context optimization page, the full method and failure cases in the benchmark writeup, and the algorithm itself open source as barber-llm.

Lever four: bound the output

Output tokens usually cost several times more than input tokens, so an unbounded response is the most expensive kind of waste. Cutting output tokens is often the cleanest win on the bill, and it comes from a few habits: set a sane max-tokens ceiling, ask for terse structured answers instead of prose, and watch two silent inflators. Structured JSON output carries a token tax from the syntax itself, and extended thinking or reasoning tokens are billed as output even though the user never sees them, so leaving reasoning on by default for easy requests quietly multiplies their cost.

Estimated saving: 10 to 30% of output cost.

Lever five: batch what does not need to be real time

Any workload that is not user-facing in real time (evaluations, enrichment, backfills, nightly summarization) should go through the batch API, where OpenAI and Anthropic both offer roughly a 50% discount in exchange for asynchronous completion within a time window. This is a pure config change for the eligible traffic, with no quality tradeoff, and most teams have more batch-eligible work than they think once they separate it from the interactive path.

Estimated saving: up to 50% on batch-eligible traffic.

The stacked math

The point of ordering these is that they compound on the same requests. A worked example on a workload that starts at $10,000 a month:

That is a 74% reduction, and none of it required a worse model on the hard requests, because the router kept the frontier available for the traffic that needed it. Your real numbers will differ, but the shape holds: the levers multiply, and routing does the heaviest lifting.

What routing looks like in code

The reason routing is the highest-leverage lever is also that it is the lowest-friction to adopt. A routing gateway that speaks the OpenAI API format is a base-URL change, not a rewrite:

import openai

client = openai.OpenAI(
    base_url="https://api.getnadir.com/v1",
    api_key="ndr_...",
)

response = client.chat.completions.create(
    model="auto",  # cheapest model that can handle this specific request
    messages=[{"role": "user", "content": prompt}],
)

Nadir can use the same OpenAI-compatible call shape for hosted execution, or return a decision beside the gateway you already run. Start in shadow mode, declare the comparison baseline, and attach actual outcomes. Complete non-streaming responses can optionally enter the verifier; streaming bypasses it, and decision-only projected savings stay advisory.

Where to start this week

• Instrument first. If you cannot break cost down by feature, do that before anything else. Every lever below is invisible without it.
• Turn on prompt caching. It is a config change with no quality cost and it pays back immediately on any cache-heavy workload.
• Put a verified router in front of your traffic. This is the largest lever and the one with the OpenAI-compatible on-ramp, so it is the highest return for the least integration work.
• Bound output and audit the system prompt. An afternoon of trimming the prefix and setting a max-tokens ceiling is often worth more than a week of anything else.
• Move non-interactive work to batch. Separate the eval and backfill traffic and take the discount.

Conclusion

There is no one switch, and anyone selling you one is selling a cap, not a cut. A spending cap limits how many requests run, not what each one costs, and it fires after the model already billed at full price. The bill drops when the unit cost of the next request drops, and that comes from the stack: measure, route, cache, trim, bound, batch. Routing is the largest lever because it is the only one that changes the price of a request before it is sent. The rest discount what is left. Applied in order, on the same traffic, they routinely more than halve a bill without a worse answer on the work that mattered.

Related reading

• LLM Routing Explained: How AI Model Routing Cuts Costs
• RouteLLM found 74% of GPT-4 calls did not need GPT-4
• Fine-tuning vs routing: the hidden cost that decides it
• Prompt caching is close to free money
• 98% of FinOps teams manage AI spend. Most cannot see where the tokens go.
• Can you speed up text? A third of input tokens removed, no answers lost

This guide links first-party Nadir benchmarks and cited third-party research throughout. The 60% / ~98% figure is a reference-assisted research ceiling on 11,420 RouterBench triples, not deployed performance or a typical customer outcome.