The Cap Is Not a Budget

Google found that cutting Gemini 3.8 Flash's maxOutputTokens to 300 barely changed thinking on easy tasks and broke 7 of 25 hard tasks. It also cost slightly more. The output cap is a ceiling, not a budget. In our illustrative model of 100,000 tasks, a capped run with retries costs 17% more than doing nothing, low thinking cuts 42%, and a Flash-Lite cascade with a cheap check lands 72% under default thinking at 98.9% solved. Includes a cross-provider table for max_tokens and thinking settings, a seven-step routing recipe, and the Python model.

Published 2026-11-04 by Dor Amir on the Nadir blog.

Filed under Routing & Cascades.

The short version.

On November 3, Google published a small routing experiment with a result that contradicts a common instinct. Lowering `maxOutputTokens` did not make Gemini 3.8 Flash think less. With a 300-token cap, easy tasks used almost the same thinking tokens as with a 4,096-token cap and cost slightly more. Hard tasks got worse: the cap broke 7 of 25, six of them because the model ran out of room mid-reasoning and returned no answer.

The output cap is a ceiling, not a budget. The settings that change the bill are which model you call and how much it is allowed to think, and the measure that tells you whether they worked is cost per successful task, not cost per token. This post explains why the cap fails, rebuilds the comparison in a small cost model you can run, and ends with a routing recipe for both settings at once.

Google's numbers come from Google's post, cited at the end. Everything else is our own illustrative model at Gemini list prices retrieved November 4, 2026. It is not measured production traffic, and nothing here comes from Nadir customer data.

Animated diagram: a task goes to a cheap classifier that asks one yes/no question, whether the task needs more than one sequential step. No routes to a small model with no thinking, yes routes to a mid model with thinking set to low. Below, the maxOutputTokens setting is struck through as a ceiling that does not change the bill.
Animated diagram: a task goes to a cheap classifier that asks one yes/no question, whether the task needs more than one sequential step. No routes to a small model with no thinking, yes routes to a mid model with thinking set to low. Below, the maxOutputTokens setting is struck through as a ceiling that does not change the bill.

What Google tested.

Sergio Villani of Google Cloud ran 50 auto-graded coding-agent questions against live Gemini models, with difficulty labeled by hand. The configurations were Gemini 3.8 Flash at default and low thinking, Gemini 3.5 Flash-Lite with no thinking, and Gemini 3.1 Pro. A Jev decision model routed between them. The findings worth taking with you:

FindingGoogle's result
Strongest setup without a router3.8 Flash on low thinking, 50 of 50 solved
Flash-Lite aloneEvery simple task in about half a second, 6 of 25 hard tasks missed
300-token output cap, easy tasksThinking tokens almost unchanged, slightly higher cost
300-token output cap, hard tasks7 of 25 broken: 6 empty, 1 wrong
Keyword and length heuristic router93.8% on its 16 dev tasks, 70.6% on 34 unseen ones
Classifier router with one atomic yes/no question34 of 34 held out, 48 of 50 overall
Router: Flash-Lite, escalate to Flash on low50 of 50

Google is clear about the limits: one run per setup, 50 prompts, simple grading. On prompts this short, the router's saving over low thinking everywhere was small per task. The post's central line is the part that generalizes: "If your cheap option doesn't actually make the model think less on easy tasks, no router can save you money."

Two other details are easy to miss. The first attempt at a classifier asked one broad simple-or-complex question and sent almost everything to simple. It started working when the question became one atomic factor: does this task need more than one sequential calculation, simulation step, or deduction? The second is that the keyword heuristic lost 23 points on unseen tasks. Rules written on your dev set tend to fit your dev set.

Why the cap does not cut thinking.

maxOutputTokens on Gemini, max_output_tokens on OpenAI, and max_tokens on Anthropic all do the same thing: stop generation once the count is reached. Thinking tokens count toward that limit on all three, and they bill at the output rate. Google's pricing page labels every Gemini output price "including thinking tokens."

A model does not plan its reasoning around the ceiling. An easy task that needs 120 thinking tokens uses 120 under a 300 cap and 120 under a 4,096 cap. The cap never binds, so it never saves anything. A hard task that needs 1,800 tokens hits the ceiling mid-thought. You are billed for everything it generated before the cut, and you get either no answer or a rushed one.

ProviderThe ceilingThe setting that changes thinkingWhat hitting the ceiling looks like
Google GeminimaxOutputTokensthinkingLevel (for example LOW)Truncated or empty answer, thinking still billed
OpenAImax_output_tokensreasoning.effortResponse marked incomplete, reasoning tokens still billed
Anthropic Claudemax_tokenseffort, or a thinking budget below max_tokensstop_reason of max_tokens, output billed up to the cut

We compared the three thinking settings themselves in The Overthinking Tax. This post covers the setting people reach for instead, and why it doesn't work.

The model.

We simulated 100,000 tasks at Gemini list prices: 3.8 Flash at $0.75 input and $3.75 output per million tokens, 3.5 Flash-Lite at $0.30 and $2.50. Every prompt is 1,200 tokens. Sixty percent of tasks are easy and forty percent are hard. Thinking tokens per task are drawn from a lognormal distribution:

AssumptionEasy taskHard task
Thinking, default level (median)1201,800
Thinking, low level (median)60700
Visible answer60300
Flash-Lite success, no thinking98%76%
Flash success, either level100%100%

The 76% figure mirrors Google's 6 of 25 missed hard tasks. Our hard tasks think longer than Google's short prompts did, so in our model the 300-token cap truncates every hard task instead of 28% of them. That is the point of the exercise: the longer your hard tasks reason, the more the cap breaks.

Histogram of output tokens per task at default thinking, log scale. Easy tasks cluster between about 100 and 300 tokens, almost all left of the 300-token cap line. Hard tasks cluster between about 700 and 8,000, all to the right of it.
Histogram of output tokens per task at default thinking, log scale. Easy tasks cluster between about 100 and 300 tokens, almost all left of the 300-token cap line. Hard tasks cluster between about 700 and 8,000, all to the right of it.

Nearly the whole easy distribution sits below the line. Only 2.3% of easy tasks reach the cap. All of the hard ones do.

Results.

Configuration$ per 1,000 tasksSolved$ per 1,000 solved
Flash, default thinking$5.00100.0%$5.00
Flash, default + 300 cap, retry on truncation$5.84100.0%$5.84
Flash, default + 300 cap, no retry$1.7758.5%$3.03
Flash, low thinking$2.91100.0%$2.91
Flash-Lite only$0.7589.1%$0.84
Classifier router (96% accurate): Lite or Flash-low$2.3498.5%$2.38
Cascade: Lite, cheap check, escalate to Flash-low$1.3698.9%$1.38
Bar chart of USD per 1,000 successful tasks: default thinking plus cap with retry $5.84, default thinking $5.00, low thinking $2.91, classifier router $2.38, cascade with check $1.38, Flash-Lite only $0.84 at 89.1 percent solved.
Bar chart of USD per 1,000 successful tasks: default thinking plus cap with retry $5.84, default thinking $5.00, low thinking $2.91, classifier router $2.38, cascade with check $1.38, Flash-Lite only $0.84 at 89.1 percent solved.

The cap with retries costs 17% more than doing nothing. Every truncated attempt is billed and then paid for again in the retry. Without a retry, the capped run looks cheap per attempt because 41.5% of tasks return nothing.

Turning thinking down is the first saving, and it needs no router. Low thinking cuts the bill 42% against default thinking, with every task still solved in this model, and Google saw the same thing with real calls. That makes it the baseline any router has to beat, which was Google's recommendation too.

Routing on both settings goes further. The classifier router lands at $2.38 per 1,000 solved, 18% under low thinking everywhere. A cascade that tries Flash-Lite first, runs a cheap check, and escalates the failures lands at $1.38, 53% under low thinking and 72% under the default. The cost is quality. The router misses 1.5% of tasks through misroutes, and the cascade misses 1.1%, which are the failures its check didn't catch.

Scatter plot of tasks solved against USD per 1,000 tasks attempted. The cascade and the router sit just below 100 percent at the left of the quality band. Low thinking sits at 100 percent at $2.91. Default thinking and the capped retry sit at the right. Flash-Lite only sits at 89 percent and the capped run without retry at 58.5 percent.
Scatter plot of tasks solved against USD per 1,000 tasks attempted. The cascade and the router sit just below 100 percent at the left of the quality band. Low thinking sits at 100 percent at $2.91. Default thinking and the capped retry sit at the right. Flash-Lite only sits at 89 percent and the capped run without retry at 58.5 percent.

Cost per success can still mislead. Flash-Lite alone has the lowest cost per solved task, at $0.84, and solves only 89.1%. A wrong answer that ships has a cost the metric does not count. So set a quality floor first, then compare cost per success among the configurations that clear it.

The recipe.

  1. Delete the cap as a cost control. Keep a generous max_tokens as a safety limit against runaway generation. Stop expecting it to save money.
  2. Set the thinking level explicitly. Most call sites run at the vendor default. Measure low against default on a few hundred real tasks. If quality holds, take the saving.
  3. Ask one atomic routing question. "Does this need more than one sequential step?" beats "is this hard?" A broad question collapses to one answer, and keyword rules overfit. See Jev, Laya, GLiNER: Beaten by len() for how cheap routing signals behave on held-out data.
  4. Route on model and thinking level together. Easy tasks go to a small model with no thinking. Multi-step tasks go to a mid model with low thinking. Reserve the frontier model and high thinking for what fails both.
  5. Check before you escalate. A schema match, a unit test, or a cited span catches most cheap-model failures for a fraction of a call. A failure nobody detects is the expensive kind. See The Retry Tax.
  6. Keep long context on one model. Google points out that a cache belongs to one model. Switching models every turn on a long agent session either needs two caches or loses the cache discount. Route short, self-contained tasks, and keep long, context-heavy turns where their cache lives.
  7. Report cost per successful task, with a quality floor. Log thinking tokens, truncations (stop_reason, finish_reason, incomplete status), retries, and escalations. Divide spend by tasks that passed your check.

The model, in code.

import numpy as np
rng = np.random.default_rng(7)
PRICE = {"flash": (0.75, 3.75), "lite": (0.30, 2.50)}   # $/M in, out (thinking included)
N, IN = 100_000, 1200
easy = rng.random(N) < 0.6
ln = lambda med, s: rng.lognormal(np.log(med), s, N)
think_def = np.where(easy, ln(120, 0.35), ln(1800, 0.6))
think_low = np.where(easy, ln(60, 0.5), ln(700, 0.6))
ans = np.where(easy, 60, 300)
lite_ok = rng.random(N) < np.where(easy, 0.98, 0.76)
usd = lambda m, i, o: (i * PRICE[m][0] + o * PRICE[m][1]) / 1e6

default = usd("flash", IN, think_def + ans)
trunc = think_def + ans > 300                          # the 300-token cap binds
capped_retry = np.where(trunc, usd("flash", IN, 300) + default, default)
low = usd("flash", IN, think_low + ans)
lite = usd("lite", IN, ans)
caught = ~lite_ok & (rng.random(N) < 0.90)              # check catches 90% of misses
cascade = lite + usd("lite", 300, 20) + np.where(caught, low, 0)

for name, c, ok in [("default", default, 1.0), ("cap+retry", capped_retry, 1.0),
                    ("low", low, 1.0), ("cascade", cascade, (lite_ok | caught).mean())]:
    print(f"{name:10s} ${c.mean()*1000:.2f}/1k tasks  solved {ok:.1%}")

This prints about $5.00, $5.84, $2.91, and $1.36 per 1,000 tasks. Replace the medians, the 60/40 split, and the success rates with numbers from your own logs before you trust any of it.

Where Nadir fits.

This is the decision a routing layer is built to make on every request. Nadir's gateway takes model="auto" and picks a model per request by classifying the task. Its observe mode records what it would have routed without changing anything, so you can see your own easy-to-hard mix, and what thinking and truncation cost you, before you switch any traffic. Savings depend on your workload, and nothing in this post forecasts yours.

Get a free key, or estimate savings from your current spend.

Limitations.


Sources: [Google for Developers Blog, "Cost per task: using cheap classifiers to route across Gemini models and thinking levels," Sergio Villani, November 3, 2026](https://developers.googleblog.com/cost-per-task-using-cheap-classifiers-to-route-across-gemini-models-and-thinking-levels/). [Google AI Studio, Gemini API pricing](https://aistudio.google.com/docs/pricing), retrieved November 4, 2026: Gemini 3.8 Flash $0.75 input and $3.75 output per million tokens through December 31, 2026, Gemini 3.5 Flash-Lite $0.30 and $2.50, output prices including thinking tokens. [OpenAI, Reasoning models](https://platform.openai.com/docs/guides/reasoning). [Claude API Docs, Effort](https://platform.claude.com/docs/en/build-with-claude/effort). Charts were generated with Python and seaborn from the illustrative model above.

More on routing & cascades

Pages on this site

Machine-readable summaries of this site: llms.txt and llms-full.txt.