The bill that arrives three times the forecast The most common way teams first take LLM costs seriously is the same story every time. The API invoice lands at two or three times what the model was supposed to cost, finance asks which feature spent it, and nobody can answer, because the tokens were never attributed to anything. There is no single setting that fixes this. There is a stack of six levers, and the important thing about them is that they multiply. Cut input tokens 30% and route half your traffic to a model a fifth of the price, and you have not saved 30%, you have saved much more, because the discounts compound on the same requests. This is the complete guide to the six levers, ordered the way you should actually apply them: measurement first, then the changes ranked by return on effort. For each one you get the realistic savings range and a link to the deep dive if you want the mechanics. Lever zero: you cannot cut what you cannot see Before any optimization, you need per-feature cost attribution. Not a monthly total, a breakdown: cost per endpoint, per user action, per agent run. The reason is that LLM spend is almost never uniform. A single feature, a single retry loop, a single oversized system prompt usually accounts for the majority of a surprising bill, and you cannot find it from an aggregate number. The FinOps discipline caught up to this in 2026: 98% of FinOps teams now manage AI spend, and most still cannot see where the tokens go. The fix is boring and non-negotiable. Log tokens in and out, model, and cost on every call, tagged with the feature that made it. Datadog's own telemetry found 69% of tokens were going into system prompts that most teams never inspected. You will not believe where the money goes until you measure it, and you will not measure it by accident. Once you can see the bill, apply the levers below in order. Lever one: route each request to the cheapest capable model This is the largest single lever, and the one most teams skip, because the default behavior of every SDK is to send every request to whatever model you hard-coded, usually the best and most expensive one. But most requests are not hard. Meta's RouteLLM work found 74% of GPT-4-class calls did not need a GPT-4-class model to get the same answer. You are paying frontier prices for a workload that is mostly easy. LLM routing is a decision layer that reads each request, estimates its difficulty, and sends it to the cheapest model that can still answer it well. The critical detail is verification. A router that guesses the model before seeing an answer is dead reckoning, and it ships bad responses when it guesses wrong. A verified router lets the cheap model answer, scores that answer with a calibrated verifier, and escalates only on a reject. That distinction is the difference between routing that costs you quality and routing that does not. On 11,420 RouterBench held-out triples, a verified router cut cost 60% against always using the top model while preserving about 98% of its quality. Routing is also strictly better than the alternatives teams reach for first: it beats fine-tuning on total cost for most workloads, because there is no training run and no endpoint to host, and it beats a blanket swap to a cheaper model, because swapping every request to a model a tenth the price breaks on the hard tail, which is exactly where a router keeps the frontier in play. Estimated saving: 40 to 60% on a mixed workload. Lever two: stop resending the same tokens uncached Agents and chatbots resend the same large prefix on every turn: the system prompt, the tool schemas, the retrieved documents, the conversation so far. Provider prompt caching lets you pay a steep discount on that repeated prefix instead of full price every time. OpenAI and Anthropic both discount cached input tokens sharply, and for a workload with a big stable prefix that is close to free money left on the table. Two adjacent techniques stack on top. Semantic caching returns a stored answer when a new request is similar enough to a past one, skipping the model call entirely. And on self-hosted fleets, routing across the KV cache instead of blind round-robin avoids recomputing context that already lives on a replica, a gap that produced a measured 170x swing in time-to-first-token on identical hardware. Estimated saving: 30 to 50% of input cost on cache-heavy workloads. Lever three: cut the input tokens you actually send Caching discounts the tokens you resend. This lever removes tokens you never needed to send at all. The usual offenders: System prompt bloat. Prompts accrete instructions nobody removes. Auditing and trimming the system prompt is often the highest-return hour of work available, because that prefix is on every single request. Tool and function schemas. MCP tool schemas and function-calling definitions are pure overhead sent on every call. Prune the tools a given request can actually use. Over-retrieval in RAG. Stuffing twenty chunks when three would do is paying for retrieval you do not use. And long context windows are not free just because they fit: a model degrades on a bloated context well before its advertised limit, so you pay more for a worse answer. Prompt compression. Tools like LLMLingua compress the prompt itself while preserving meaning, trading a little preprocessing for fewer input tokens. Estimated saving: 15 to 40% of input cost. Lever four: bound the output Output tokens usually cost several times more than input tokens, so an unbounded response is the most expensive kind of waste. Cutting output tokens is often the cleanest win on the bill, and it comes from a few habits: set a sane max-tokens ceiling, ask for terse structured answers instead of prose, and watch two silent inflators. Structured JSON output carries a token tax from the syntax itself, and extended thinking or reasoning tokens are billed as output even though the user never sees them, so leaving reasoning on by default for easy requests quietly multiplies their cost. Estimated saving: 10 to 30% of output cost. Lever five: batch what does not need to be real time Any workload that is not user-facing in real time (evaluations, enrichment, backfills, nightly summarization) should go through the batch API, where OpenAI and Anthropic both offer roughly a 50% discount in exchange for asynchronous completion within a time window. This is a pure config change for the eligible traffic, with no quality tradeoff, and most teams have more batch-eligible work than they think once they separate it from the interactive path. Estimated saving: up to 50% on batch-eligible traffic. The stacked math The point of ordering these is that they compound on the same requests. A worked example on a workload that starts at $10,000 a month: | Step | Lever | Applied to | Running bill | |---|---|---|---| | Start | none | | $10,000 | | 1 | Route 60% of traffic to a model ~5x cheaper | all interactive | $5,200 | | 2 | Cache the repeated prefix | cache-heavy input | $3,900 | | 3 | Trim system prompt and tool schemas | all input | $3,300 | | 4 | Bound output length | all output | $2,900 | | 5 | Move evals and backfills to batch | ~15% of volume | $2,600 | That is a 74% reduction, and none of it required a worse model on the hard requests, because the router kept the frontier available for the traffic that needed it. Your real numbers will differ, but the shape holds: the levers multiply, and routing does the heaviest lifting. What routing looks like in code The reason routing is the highest-leverage lever is also that it is the lowest-friction to adopt. A routing gateway that speaks the OpenAI API format is a base-URL change, not a rewrite: import openai client = openai.OpenAI( base_url="https://api.getnadir.com/v1", api_key="ndr_...", ) response = client.chat.completions.create( model="auto", # cheapest model that can handle this specific request messages=[{"role": "user", "content": prompt}], ) Same call shape as any OpenAI-compatible endpoint. Nadir reads the prompt, routes to the cheapest capable model, and a calibrated verifier scores the cheap answer before it ships, escalating on a reject instead of returning a bad response. The savings dashboard shows the real delta against always-Opus, per request, so lever one stops being a guess and starts being a measured line on the bill. Where to start this week Instrument first. If you cannot break cost down by feature, do that before anything else. Every lever below is invisible without it. Turn on prompt caching. It is a config change with no quality cost and it pays back immediately on any cache-heavy workload. Put a verified router in front of your traffic. This is the largest lever and the one with the OpenAI-compatible on-ramp, so it is the highest return for the least integration work. Bound output and audit the system prompt. An afternoon of trimming the prefix and setting a max-tokens ceiling is often worth more than a week of anything else. Move non-interactive work to batch. Separate the eval and backfill traffic and take the discount. Conclusion There is no one switch, and anyone selling you one is selling a cap, not a cut. A spending cap limits how many requests run, not what each one costs, and it fires after the model already billed at full price. The bill drops when the unit cost of the next request drops, and that comes from the stack: measure, route, cache, trim, bound, batch. Routing is the largest lever because it is the only one that changes the price of a request before it is sent. The rest discount what is left. Applied in order, on the same traffic, they routinely more than halve a bill without a worse answer on the work that mattered. Related reading LLM Routing Explained: How AI Model Routing Cuts Costs RouteLLM found 74% of GPT-4 calls did not need GPT-4 Fine-tuning vs routing: the hidden cost that decides it Prompt caching is close to free money 98% of FinOps teams manage AI spend. Most cannot see where the tokens go. This guide links first-party Nadir benchmarks and cited third-party research throughout. Savings ranges are typical outcomes on mixed production workloads and will vary with your traffic mix. The 60% cost reduction with ~98% quality figure is measured on 11,420 RouterBench held-out triples against an always-top-model baseline.