Own the Base, Rent the Peak

TL;DR.

Self-hosting looks like the obvious answer to an AI bill that keeps climbing. In October 2026 the math says otherwise, for most teams and most traffic.

• A rented H100 at 100% utilization costs about $0.139 per 1M blended tokens for a 20B-class model, using the assumptions below. Claude Haiku 5.5 lists at $0.136. A GPU that is busy every second of the month still doesn't beat the cheapest APIs.
• Real traffic doesn't keep a GPU busy. Size a fleet for the weekday peak and our synthetic week runs it at 32%. At that utilization each token costs about 3x the full-load price.
• Against mid-priced APIs, self-hosting does win, at a known utilization. The break-even is 29% against Qwen Plus, 36% against DeepSeek V4.1-Flash at peak, and 14% against Gemini 3.8 Flash.
• The cheapest self-hosted setup is hybrid. Own the baseline, rent the peak. On the same week, 5 GPUs plus API overflow cost $14,054 a month, against $20,075 for 11 peak-sized GPUs and $21,875 for an API alone.
• And the biggest lever isn't hosting at all. The same traffic on Claude Sonnet 5.5 costs $126,202. Picking the right tier moves the bill 20x. Choosing where to host it moves it by about 30%.

Below: the break-even formula, a table you can check against your own GPU quote, a 60-line Python tutorial that finds the cheapest GPU count for your traffic, and where a router fits.

Every GPU figure here is a model with stated assumptions, not a measurement: $2.50 per H100-hour and 5,000 blended tokens per second at full batch. API prices are October 2026 list prices from our [pricing comparison](/blog/llm-api-pricing-june-2026-complete-comparison). The traffic week is synthetic, not Nadir customer data.

Why everyone is asking this now.

Two things happened this quarter. Companies started rationing AI spend: Microsoft reportedly cut internal AI allowances, Meta reportedly cut its Claude Code usage, and Uber reportedly burned its annual AI budget in four months. And developers hitting subscription limits started pointing their coding agents at local models through Ollama, LM Studio and llama.cpp.

So "should we just run it ourselves?" is in a lot of planning docs right now. Back in July we covered the research on why GPU utilization sets the self-hosted price. This post is the practical version, priced against October's rate cards. Those rate cards changed the answer, because the cheapest API tier fell to about a dime per million input tokens.

The only formula you need.

A rented GPU bills by the hour whether it serves tokens or not. So the cost per token is:

cost per 1M tokens = GPU $/hour / (tokens per second x 3,600 / 1,000,000) / utilization

At $2.50 an hour and 5,000 tokens per second, one GPU serves 18M tokens an hour. That's $0.139 per 1M at full load. At 50% utilization it's $0.278. At 10% it's $1.39.

API prices need one adjustment to compare: blend input and output at your real ratio. Agent and RAG traffic is input-heavy, so we use 10 input tokens per output token:

blended API price = (10 x input price + output price) / 11

Divide the GPU's full-load price by an API's blended price and you get the break-even utilization: the share of the month your GPU has to be busy to match that API.

The break-even table.

The last row is the trap in most "self-hosting saves 90%" posts. A 20B model on your GPU beats Sonnet 5.5 on price at almost any utilization, but it isn't a Sonnet 5.5. It replaces only the traffic a small model can serve. That's a routing question, not a hosting one. We'll come back to it.

The assumptions move the table, so here's how much:

Two things stand out. Throughput matters as much as the GPU quote, and you control it: batch size, quantization, prefix caching and context length all move it. Measure it with vllm bench serve on your own prompts before you trust any number here, ours included. And the cheapest APIs stay hard to beat. Even on a $1.49 marketplace GPU, you need the GPU busy 61% of the month to match Haiku 5.5.

Peaks set the size. Nights set the bill.

Utilization isn't a setting. It's what your traffic does to your fleet. Most production traffic has a weekday peak, quiet nights and quieter weekends. A fleet sized to serve the peak without queueing sits mostly idle the rest of the week.

In our synthetic week, peak traffic is 2.9x the average. Covering the peak takes 11 GPUs, and those 11 GPUs run at 32% for the month. At 32%, the effective price is $0.43 per 1M. That's above every cheap API in the table and only just below Qwen Plus.

The fix is to stop sizing for the peak. Size the fleet for the volume you carry most of the week, run it hot, and send what doesn't fit to an API that bills per token. The GPUs only ever see the volume they can keep busy. The API only sees the spikes, which is the traffic per-token billing handles best.

Tutorial: find your cheapest GPU count.

The script below takes a CSV of hourly token volume and answers three questions:

1. What does a token cost on your GPUs at your real utilization?
2. At what utilization does each API become cheaper?
3. How many GPUs minimize the monthly bill, if overflow goes to an API?

Step 1: export hourly token volume.

You want one row per hour, for at least a full week, so weekends count. Use only the traffic a small model can serve. If you route already, that's the share your router sends to the small tier. If you don't, the next section covers how to find it.

hour,tokens
0,31204551
1,29887310
2,30115922

Any gateway log or provider usage export gets you there with a GROUP BY on the hour. Count input and output tokens together. The blend ratio is handled separately.

Step 2: the script.

"""idle_gpu.py: what does a self-hosted model cost per token, and when should
overflow go to an API?

Input: CSV of hourly token volume (hour,tokens) for traffic a small model
can serve. Prints effective $/1M at your real utilization, break-even
utilization against each API, and the cheapest GPU count for a hybrid setup.
"""
import csv, math, sys

GPU_HOUR = 2.50          # $ per H100-hour, on-demand rental (your quote here)
TOKENS_PER_SEC = 5_000   # blended in+out tokens/s per GPU at full batch (measure it)
OPS_PER_MONTH = 0        # on-call, upgrades, evals; 0 flatters self-hosting
IN_OUT = 10              # input tokens per output token in your traffic

# $ per 1M (input, output), October 2026 list prices
APIS = {
    "Claude Haiku 5.5":           (0.10, 0.50),
    "GPT-OSS 20B (Groq)":         (0.075, 0.30),
    "DeepSeek V4.1-Flash, peak":  (0.30, 1.20),
    "Qwen Plus":                  (0.40, 1.20),
    "Gemini 3.8 Flash":           (0.75, 3.75),
    "Claude Sonnet 5.5":          (2.00, 10.00),
}
FALLBACK = "Qwen Plus"   # the API your overflow would go to

def blended(p_in, p_out):
    return (IN_OUT * p_in + p_out) / (IN_OUT + 1)

hours = [float(r["tokens"]) for r in csv.DictReader(open(sys.argv[1]))]
month = 730 / len(hours)                     # scale the sample to a month
cap = TOKENS_PER_SEC * 3600                  # tokens one GPU serves per hour
full = GPU_HOUR / (cap / 1e6)                # $/1M at 100% utilization
total = sum(hours) * month

print(f"traffic {total/1e9:.2f}B tokens/month, peak/avg {max(hours)/(sum(hours)/len(hours)):.1f}x")
print(f"self-hosted at 100% utilization: ${full:.3f} per 1M\n")
print(f"{'API':<28}{'$/1M':>7}{'break-even util':>17}")
for name, p in APIS.items():
    b = blended(*p)
    be = full / b
    print(f"{name:<28}{b:>7.3f}{('never' if be > 1 else f'{be:.0%}'):>17}")

api = blended(*APIS[FALLBACK])
def plan(gpus):
    local = sum(min(h, gpus * cap) for h in hours) * month
    spill = total - local
    fixed = gpus * GPU_HOUR * 730 + (OPS_PER_MONTH if gpus else 0)
    return fixed + spill / 1e6 * api, local / (gpus * cap * len(hours) * month) if gpus else 0, spill / total

peak = math.ceil(max(hours) / cap)
best = min(range(peak + 1), key=lambda g: plan(g)[0])
print(f"\n{'plan':<24}{'GPUs':>5}{'monthly':>10}{'GPU util':>10}{'to API':>8}")
for label, g in [("all API", 0), ("all local, peak-sized", peak), ("hybrid, cheapest", best)]:
    c, u, s = plan(g)
    print(f"{label:<24}{g:>5}{c:>10,.0f}{u:>10.0%}{s:>8.0%}")

The model is deliberately simple. Each hour, the GPUs serve up to their capacity and the rest spills to the fallback API. It ignores queueing, so a real fleet needs some headroom below 100%. It also ignores latency SLOs, which usually force more headroom still. Both push the real answer toward fewer GPUs and more API, not the other way.

Step 3: read the output.

On the synthetic week, 46.3B tokens a month, with Qwen Plus as the fallback:

traffic 46.27B tokens/month, peak/avg 2.9x
self-hosted at 100% utilization: $0.139 per 1M

API                            $/1M  break-even util
Claude Haiku 5.5              0.136            never
GPT-OSS 20B (Groq)            0.095            never
DeepSeek V4.1-Flash, peak     0.382              36%
Qwen Plus                     0.473              29%
Gemini 3.8 Flash              1.023              14%
Claude Sonnet 5.5             2.727               5%

plan                     GPUs   monthly  GPU util  to API
all API                     0    21,875        0%    100%
all local, peak-sized      11    20,075       32%      0%
hybrid, cheapest            5    14,054       55%     23%

How to read it:

• Peak-sized local barely beats the API. 11 GPUs at 32% come out $1,800 a month under Qwen Plus. Add one engineer's share of on-call through OPS_PER_MONTH and it loses.
• Hybrid wins by 30 to 36%. Five GPUs run at 55% and the API takes 23% of tokens, all of them from weekday peaks. The bill is flat across 3 to 6 GPUs ($14,054 to $14,334), so you can pick the count your on-call rotation is comfortable with.
• Change the fallback and the answer moves. With DeepSeek V4.1-Flash at peak as the fallback, the best plan is 3 GPUs at 70% for $12,630. With Gemini 3.8 Flash it's 8 GPUs for $16,608. With Haiku 5.5 it's zero GPUs: the whole month costs $6,310 on the API.

That last line is the honest summary. If the cheapest API that clears your quality bar is a dime-per-million model, you shouldn't self-host for cost. Self-host for data residency, for a model you've fine-tuned, or for predictable spend. Then go hybrid, so the GPUs you do buy stay busy.

The lever that's 20x bigger.

Every plan above assumes the traffic is already small-model traffic. In most stacks it isn't labeled that way. It's mixed into the same model= string as the hard prompts, and all of it bills at the frontier rate.

The gap between the top bar and the rest is the real decision. $126,202 against $6,310 to $21,875 is a 6 to 20x swing from sending each request to a tier that can handle it. The gap between hosting choices, $14,054 against $20,075, is about 30%. Both are worth having. The order matters: route first, then decide where the small tier runs.

To find your small-model share:

1. Sample a few hundred real requests from a week of logs.
2. Replay them on the small model you'd host, or its API twin.
3. Grade both answers with a rubric or a judge, and count the share where the small model is good enough.
4. Use that share as the traffic in Step 1, not your total volume.

We walk through this kind of replay in shadow mode, forever, and the case for small models in agent loops in the 7B substitution.

What about the laptop?

The other half of this quarter's local-model wave is developers running coding agents against Ollama on their own machines. The math there is different: you already own the GPU, so idle time costs nothing. What limits a laptop is capability. Agents need tool calling and long context (Claude Code wants 32K at a minimum, and 64K is better), and small local models still lose track of edits across many files.

The pattern that works is the same one as above, at desk scale. Let the local model take the high-volume, low-judgment turns: file reads, grep summaries, boilerplate, commit messages. Send the planning and the multi-file edits to a frontier model. That keeps the subscription or API meter for the turns that need it.

Five rules before you buy GPUs.

1. Compare against the cheapest API that clears your quality bar, not the one you use today. In October 2026 that's often a $0.10 model.
2. Measure throughput on your own prompts. Halving tokens per second doubles every break-even in this post.
3. Size for the baseline, not the peak. Send the peaks to an API. Utilization is the whole game.
4. Put ops in the model. OPS_PER_MONTH = 0 is the most flattering number in the script.
5. Route before you host. A router decides what the small tier gets. Hosting only decides what that tier costs.

Where Nadir fits.

Rules 3 and 5 are what Nadir does at the gateway. Point your OpenAI compatible base URL at Nadir and send model="auto". Each request goes to the cheapest model you've configured that clears the quality floor you set per API key, so easy traffic stops billing at the frontier rate.

A self-hosted model is just another tier. Serve open weights behind an OpenAI compatible endpoint with vLLM, TGI or Ollama, and register it next to your hosted providers. Put an API model behind it in your failover chain, so a local server that goes down isn't an outage. Every response reports which model answered and what it cost, so the hourly CSV for this tutorial builds itself, and you can see how much of your traffic is small-tier before you buy a GPU.

If your prompts can't leave your network at all, Nadir also runs self-hosted, inside your own perimeter, against your own keys and weights.

To measure before changing anything, run a week of traffic in Observe mode. It serves exactly the model you asked for and records what routing would have chosen, so your small-model share comes from your own traffic instead of our synthetic week.

Start with a free key, or see how routing works.

Sources: API list prices from our [October 2026 LLM API pricing comparison](/blog/llm-api-pricing-june-2026-complete-comparison). H100 rental range ($1.49 marketplace to $3.29 on demand per GPU-hour) from public 2026 provider listings and self-hosting guides; $2.50 is our midpoint. Throughput of 5,000 blended tokens per second is an assumption for a 20B-class model served with vLLM at full batch, not a benchmark. Reports of Microsoft, Meta and Uber cutting AI spend are from October 2026 press coverage and are not confirmed by the companies. Claude Code context guidance for local models is from community setup guides. The traffic week, the script's output and every monthly figure are synthetic and illustrative.