Abstract. Most agent cost audits ask where tokens go, then reach for the same four fixes: cache, compress, route, retrieve less. Those audits rank techniques by average savings. This post asks a sharper question, borrowed from production economics: at the margin, is the next 1,000 tokens spent on a given step worth what it costs? Using an illustrative step-level model of four token "factors" an agent funds on every step, retrieval breadth, reasoning depth, verification samples, and history retention, each with a concave, diminishing-returns quality response, this analysis finds a naive fixed-budget agent operating in four very different places relative to each factor's point of diminishing return: badly over-invested in two cheap, fast-saturating factors, one of them past the point where more tokens actively make the answer worse, and badly under-invested in the one factor with the best marginal returns left on the table. Reallocating the same total dollar budget to equalize marginal quality-per-dollar across factors, a textbook result from firm theory, recovers 45% of the quality gain available from an unconstrained optimum, at zero extra spend, while cutting total token volume 20% and per-step latency 16%. The finding: cost dashboards that track token counts, not marginal dollar returns by stage, will miss exactly this kind of free reallocation. All data, parameters, and figures in this post are illustrative and synthetic, computed from a step-level model built for this analysis. Not derived from proprietary production traces. Pricing: Claude Opus 4.8, $5/M input tokens, $25/M output tokens, Anthropic pricing as of June 2026. Full methodology and assumptions below. Why marginal, not average. Gartner's own estimate is that an agentic workflow burns 5 to 30 times more tokens per task than a single chatbot call, because a chatbot triggers one inference call and an agent triggers ten or twenty. OpenRouter's own billing data shows the consequence at platform scale: weekly token volume on the exchange went from 0.4 trillion tokens in December 2024 to 27 trillion by March 2026, a scale-up that tracks agentic adoption, not chat volume. Every cost audit responding to that growth reaches for the same playbook: cache what repeats, compress what's verbose, route what's simple, retrieve less. This blog has ranked those levers against each other by average savings before and found caching wins on average. Average savings answers "which lever is generally good." It does not answer the question an engineer actually faces mid-sprint: for the retrieval step I'm about to tune, is the next chunk I fetch worth its cost, or did I cross that line three chunks ago? That is a marginal question, not an average one, and it has a name in economics: a production function, the relationship between an input (tokens) and an output (task quality), and the point past which each additional unit of input buys less and less. A recent survey, Token Economics for LLM Agents (Chen et al., Zhejiang University and Alibaba Cloud, May 2026), makes exactly this framing explicit: it proposes treating tokens as a production factor under neoclassical firm theory, where a budget-constrained agent should allocate spend to equalize marginal return per dollar across every factor it funds, not spend a fixed, arbitrary amount on each. This post builds that model for a single agent step and checks what a naive, fixed-per-factor budget gets wrong. Research question. For a single agent step funding four token-consuming factors, retrieval breadth, reasoning depth, verification samples, and history retention, where does a naive, fixed-budget allocation sit relative to each factor's point of diminishing marginal return, and does reallocating the same total dollar budget to equalize marginal quality-per-dollar across factors improve quality, cost, or latency without any new spend? Methodology. Four factors, each modeled as tokens invested (x, in thousands of tokens) producing quality points (0–100 scale, additive, illustrative) through a concave response curve. Two factors saturate smoothly (retrieval breadth, verification samples); two rise then mildly decline past a point, modeling overthinking in extended reasoning and context rot in long, unpruned history (reasoning depth, history retention). A fixed baseline of 40 quality points represents everything else the step does regardless of these four factors. | Factor | What it buys | Price/1K tokens | Naive spend | |---|---|---:|---:| | Retrieval breadth | chunks pulled into context | $0.005 (input) | 9.0K tok | | Reasoning depth | chain-of-thought / self-critique length | $0.025 (output) | 6.0K tok | | Verification samples | independent samples scored for self-consistency | $0.025 (output) | 1.5K tok | | History retention | prior-turn tokens carried forward verbatim | $0.005 (input) | 7.0K tok | The naive allocation totals 23.5K tokens and $267.50 per 1,000 steps, chosen to resemble the kind of fixed heuristic a team ships once and rarely revisits (retrieve a fixed top-K, run a fixed reasoning budget, keep full history, run one verification pass with no self-consistency). Two reference points are computed against this baseline: Each factor's knee, the point where marginal return falls to 15% of that factor's initial marginal return. This threshold is a stated modeling choice, not a universal constant; it exists to give "diminishing return" a specific, reproducible meaning rather than an eyeballed one. A budget-optimal allocation, found by water-filling: search for the marginal-value-per-dollar cutoff λ such that funding every factor up to the point where its marginal quality-per-dollar equals λ spends exactly the naive budget's total dollars. This is the textbook equal-marginal-return condition from producer theory, restricted to each factor's first monotonically-declining branch (an allocation is never pushed onto a segment where marginal product is still rising, or has gone negative). All figures are computed from the stated model; the arithmetic is reproducible from the assumptions above. Results. Four factors, four different distances from their knee. Chart 1: Marginal quality gained per additional 1,000 tokens, by factor, with each curve's knee and the naive agent's actual spend marked. History retention's curve has already crossed below zero at the naive agent's spend point. Retrieval breadth and reasoning depth are past their knees but still producing positive returns (89.6% and 34.2% past, respectively). Verification samples sit at just 11.3% of its knee value, badly under-funded. History retention is 296% past its knee, deep enough into the decline that its curve has crossed zero: the naive agent's next history token is not neutral, it is actively harmful to answer quality, the numeric signature of context rot showing up inside a single factor's own return curve. One budget, four incompatible returns per dollar. Chart 2: Marginal quality points per additional $1 spent, at the naive agent's current spend, by factor. Verification returns +92.2 points per dollar; history retention returns -72.3. Converting marginal token returns into marginal dollar returns, since retrieval and history tokens cost a fifth of reasoning and verification tokens, sharpens the picture. At the naive agent's operating point: retrieval +35.0 quality points per dollar, reasoning +19.4, verification +92.2, history −72.3. A profit-maximizing allocation, per the equal-marginal-return condition, would never leave a −72.3 next to a +92.2 in the same budget. The naive agent does, on every single step. Matching every knee costs 78% more; reallocating the same budget captures 45% of the same gain for free. Chart 3: Cost vs. quality for naive, knee-matched-everywhere, and budget-optimal reallocation. Naive sits at $267.50/1,000 steps and 82.9 quality points; matching every factor to its knee reaches 93.4 points for $476/1,000 steps; reallocating the same $267.50 budget reaches 87.7 points. Funding every factor all the way to its own knee is the highest-quality point tested (93.4), but it costs 78% more than the naive baseline, mostly because verification's knee sits at 13.3K tokens and verification tokens are priced at output rates. Keeping the naive agent's exact budget, but reallocating it to equalize marginal dollar returns, reaches 87.7, a 5.7% quality improvement for zero additional spend, capturing 45% of the gain available from the far more expensive knee-matched allocation. Reallocation lowers latency as a side effect. Chart 4: P50 latency per step, naive vs. budget-optimal reallocation. Naive totals 3,642ms; reallocation totals 3,054ms, a 16.1% reduction, because tokens moved off the sequential retrieval and reasoning path onto parallelized verification. Retrieval and reasoning tokens sit on the critical path; verification samples, run in parallel and scored independently, barely add wall-clock time regardless of how many are drawn. Because reallocation moves tokens out of the first two factors and into the third, latency drops 16.1% without that ever being the objective the water-filling optimized for. Cost and latency point the same direction here because both are driven by which factor holds the tokens, not by a separate latency-specific intervention. The reallocated budget spends fewer tokens, not more. Chart 5: Tokens invested per factor, naive fixed budget vs. budget-optimal reallocation. Retrieval, reasoning, and history all shrink; verification more than triples. Total token volume falls from 23.5K to 18.9K, about 20%, while dollar cost stays flat. Total token volume falls from 23.5K to 18.9K tokens per step, about 20%, even though total dollar cost is unchanged. The freed tokens came from two cheap-per-token factors (retrieval, history, priced at input rates) and were reinvested into one expensive-per-token factor (verification, priced at output rates) that still had steep returns left. A cost dashboard tracking token counts as a spend proxy would report a 20% cost cut here. The bill did not move. Discussion. Average savings and marginal returns answer different questions, and most tooling only answers the first one. Ranking levers by average savings, as this blog has done before, tells a team which technique to try first. It says nothing about when to stop applying a technique that is already live, which is the more common decision: a retrieval pipeline that already runs, a reasoning budget that already ships, being quietly past their point of diminishing, or negative, return. Token volume is not a reliable proxy for dollar cost, in either direction. Section 5's result cuts both ways: token volume fell 20% while cost stayed flat, but the reverse mistake is just as available; a team funding verification samples generously and retrieval sparingly could see token counts flat while dollar cost climbs, because a "small" increase in expensive output tokens can outweigh a large decrease in cheap input tokens. Any cost-tracking system built on token counts instead of the price attached to each factor's token type will misprice exactly the tradeoffs this analysis is built to catch. Negative marginal return is a distinct failure mode from merely diminishing return. Most of the optimization literature this blog covers treats "diminishing" as the whole story: caching, compression, and retrieval precision all target factors that are still net-positive, just less efficient at the margin. History retention in this model is different: past its knee, it is actively subtracting quality, which is the numeric shape of context rot rather than ordinary waste. A factor with a negative marginal return needs a cap, not a discount. Reallocation is a free lunch only because the naive baseline was already inefficient. The 45%-of-the-gain-for-$0 result in section 3 is not a claim that reallocation always beats spending more; an agent whose naive allocation already sat near its knee on every factor would find little to reallocate. The size of the free lunch is a direct measure of how far off the naive heuristic was, which is itself useful: teams can estimate it by comparing their actual per-stage token spend against a quick knee estimate, before committing to any new infrastructure. Verification is the factor most naive agents underfund, and it is also the hardest one to add. Retrieval breadth and history length are config values; turning on self-consistency or a verifier pass is an architecture change. That asymmetry, not any lack of awareness that verification helps, is the more likely reason it sits at 11% of its knee in a realistic naive baseline: it is the one lever on this list that a team cannot flip without writing new code, which is consistent with why best-of-N and self-consistency sampling remain comparatively under-deployed relative to how much marginal value they return. Implementation framework. Step 1: Name the factors your agent actually funds per step. Not every agent has these exact four; a coding agent might fund diff generation, test execution, and file-search breadth instead. The method requires naming factors with a clear token cost and a plausible concave return, not reusing this post's four. Step 2: Measure marginal quality per 1,000 tokens for each factor, not just total tokens spent. Vary one factor's budget, holding the others fixed, and score the output against a rubric or eval set at each level. This is the same instrumentation Anthropic's token counting API supports for total-token measurement, applied per factor instead of per request. def marginal_curve(factor_name: str, budgets: list[int], run_step, score) -> list[tuple[int, float]]: results = [] for budget in sorted(budgets): output = run_step({factor_name: budget}) results.append((budget, score(output))) return results # fit a concave curve, then take its derivative Step 3: Convert every factor's curve to marginal quality per dollar, not per token. Factors billed at different rates (input vs. output, cached vs. uncached) are not comparable in token units. Section 2's chart exists because this conversion is the step naive audits skip. Step 4: Find where each factor's marginal $ return has gone negative, and cap it there immediately. This is a bug fix, not an optimization tradeoff: a factor past this point is actively hurting quality while costing money. Section 1's history-retention curve is the pattern to look for. Step 5: Water-fill the existing budget across the remaining factors. Given the current total spend, raise the marginal-value-per-dollar cutoff until funded factors converge on the same rate of return. This is a reallocation, not a spending decision, and it is the step that produced section 3's zero-cost quality gain. Step 6: Only then decide whether to raise the total budget. Section 3 showed the highest-quality point tested costs 78% more. That may be worth it for a task where quality has high value, but it is a separate decision from reallocation and should never be made before reallocation, since reallocation captures a meaningful fraction of the same gain for free. Step 7: Re-measure quarterly, or after any change to prompts, tools, or traffic mix. A concave curve fit to today's traffic will not hold as retrieval corpora grow, reasoning models change, or task mix shifts. Marginal returns are a property of a workload at a point in time, not a fixed constant. The product angle. Nothing in sections 1 through 5 is visible from a dashboard that reports total tokens or total dollars per request. Seeing it requires per-stage token counts, split by price tier, tracked over enough requests to fit a return curve, exactly the kind of continuous, per-request instrumentation most teams do not build until the bill is already a problem. This is the layer Nadir is built to be: routing decisions and cost breakdowns logged per request, by stage, so a team can see its own marginal-return curves instead of borrowing the illustrative ones in this post. The reallocation this analysis found for $0 extra spend is only actionable if a team can see, in its own traffic, which factor has gone negative and which one is still under-funded. That visibility, not a bigger discount on any single model, is what turns "we think we're overspending on retrieval" into a specific, fundable ticket. Conclusion. Average savings tell a team which lever to reach for. Marginal returns tell it when to put the lever down. A naive, fixed-budget agent is not wrong to fund retrieval, reasoning, verification, and history, it is wrong about how much of each, and the gap between its allocation and an efficient one showed up here as a 20% cut in token volume, a 16% cut in latency, and a 5.7% quality gain, all for the same dollar it was already spending. The 78%-more-expensive, highest-quality point is a real option; it is just not the first one worth taking. Measure the marginal return before the next budget. See your own agent's marginal returns, by stage →* All figures in this post are illustrative, computed from a synthetic step-level production-function model built for this analysis, not measured from proprietary production traffic. Pricing: Claude Opus 4.8 at $5/M input, $25/M output (Anthropic, June 2026). Sources: Token Economics for LLM Agents: A Dual-View Study from Computing and Economics, arXiv:2605.09104, May 2026. When More Thinking Hurts: Overthinking in LLM Test-Time Compute Scaling, arXiv:2604.10739, April 2026. Gartner, via Computer Weekly: "Agentic AI won't benefit from economies of scale". OpenRouter Series B announcement, weekly token volume data. Anthropic Claude Pricing, June 2026. Anthropic Token Counting API.*