The Invoice Won't Tell You

Abstract.

A provider invoice tells you how many input and output tokens an agent consumed. It does not tell you which stage of the workflow created them. We simulate a research agent that searches, reads pages and synthesizes an answer, then attribute every billed token to the stage that produced it, including each later resend. In the naive version, page reads account for 73% of billed tokens, final synthesis for 0.2%, and about 53% of all tokens are waste under our stated assumptions. Four architectures, from premium-everywhere to routing plus selective reads plus caching, cut modeled cost per 1,000 tasks from $3,584 to between $2,324 and $524. The practical takeaway: instrument by stage first, because the invoice cannot show you where to cut.

Every number and chart in this post comes from a simulation with assumptions listed below. It is illustrative, not measured on production traffic, and your results will differ. Prices are placeholders, and the quality scores are assumptions, not measurements.

Introduction.

Agent workloads consume far more tokens than chat. A 2026 study of agentic coding runs reports roughly 1,000x the tokens of code chat, with input tokens rather than output tokens driving the cost, and up to 30x variation between runs of the same task (Bai et al., arXiv:2604.22750). Input-dominated cost is a hint that the bill is mostly about context: what the agent carries and re-sends, not what it writes.

That is hard to act on from an invoice. "99% input tokens" does not say whether to trim the system prompt, cap page reads, deduplicate retrieval or move a step to a cheaper model. Observability standards are catching up: the OpenTelemetry GenAI conventions define gen_ai.usage.input_tokens and gen_ai.usage.output_tokens on model spans, though agent-level conventions are still marked as in development (OpenTelemetry GenAI spans). Token counts per call exist. Attribution by stage mostly does not, unless you build it.

We have covered the anatomy of one ReAct step, the decision between search and read and per-team chargeback. This post covers the layer between them: a per-stage ledger, and what it changes.

Research question.

Where do tokens get wasted in a search-and-read agent once every token is attributed to the stage that created it, and which optimizations reduce cost without a matching loss in answer quality?

Methodology.

We simulated 4,000 research tasks per architecture. Each task runs 2 + Poisson(4) search/read cycles (6.0 on average). Assumptions:

• Prices (placeholders, USD per million tokens): premium $5 in / $25 out / $0.50 cache read; small $0.25 in / $1.25 out. Cache writes cost 1.25x input. Substitute your own rate card.
• Fixed context: 900-token system prompt, 2,100 tokens of tool schemas, re-sent on every call.
• Per cycle: one query-generation call (60 output tokens), 10 search snippets at 90 tokens each, 4 pages at 3,200 tokens each, one reasoning call (450 output tokens). One synthesis call (1,200 output tokens) at the end.
• Ledger rule: every token is charged to the stage that created it, including each time it is re-sent. A page read in cycle 1 and re-sent in cycles 2 to 6 is billed to "page reads" each time.
• Waste rates (assumed): 40% of retrieved tokens irrelevant, then 15% of the remainder duplicated across cycles, 15% of the rest boilerplate, and 10% of what survives over-read. 25% of reasoning tokens repeat earlier reasoning.
• Latency (assumed): 0.4 s time to first token, 20,000 tokens/s prefill, 60 tokens/s premium decode, 150 tokens/s small decode, 1.2 s per search and 0.9 s per page read (sequential in naive, parallel in selective).
• Quality: an assumed pass rate per architecture, used only to show the shape of the tradeoff.

Four architectures: (1) naive, premium model for every call; (2) router, small model for query generation, premium for reasoning and synthesis; (3) router + triage + selective read, a small-model triage call over snippets, then reading the top 2 pages in parallel with boilerplate stripped after retrieval; (4) the same plus prompt caching, assuming a perfectly stable prefix and no cache misses, which makes it an upper bound.

Results.

Result 1: the ledger sees what the invoice cannot.

In the naive agent, 73% of billed tokens trace back to page reads, 10% to search snippets, 9% to system prompt and tool schemas, and 8% to reasoning. Query generation and final synthesis together are under 1%. The answer is the smallest line item in the bill. This agrees with the shape reported for real coding agents: cost is driven by context, not completions.

Result 2: about half the tokens are waste.

Under our assumed rates, irrelevant retrieved chunks are the largest category at 33% of billed tokens, followed by duplicate retrieval (7.5%), verbose tool output (6.3%), over-reading (3.6%) and repeated reasoning (2.0%). The ranking depends on the assumptions, but the mechanism does not: waste in a retrieved page is paid again on every later call, so it compounds. The same effect is why over-retrieval is costly in RAG pipelines.

Result 3: cost falls in steps, and each step has a different cause.

Routing only the cheap steps saves 35%, which is modest because query generation was never the expensive stage. Selective reading is the big move (a further 31 points) because it shrinks the stage the ledger flagged. Caching adds the largest relative drop, but treat 85% as a ceiling, not a forecast: it assumes every prefix hit, and the cache-write premium and TTL misses eat into it in production. Savings from routing, selection and caching also overlap, so measure them together rather than adding them.

Result 4: latency follows tokens, but routing can add hops.

Selective, parallel reads cut modeled p50 by about 18% and p95 by about 20% versus naive. Router-only p95 got worse (236 s against 214 s) in our model because an extra hop adds tail latency without removing any prefill. This matches the pattern in tail latency and parallel tool calls: speed comes from less context and fewer sequential waits, not from a cheaper model alone.

Result 5: the tradeoff has a shape.

Quality here is assumed, so read the shape only. Aggressive compression applied before retrieval removes decision-critical information and lands below hybrid routing on both axes. Small-model-first is cheapest but costs the most quality, and it ignores retries, which the retry tax can turn into a net loss.

Discussion.

• Cost cuts that skip the ledger are guesses. Routing the query-generation step was the obvious move and the smallest win.
• Too aggressive hurts. Compressing or truncating before you know what is relevant trades tokens for answer quality.
• Small models are not always cheaper. If they cause retries or escalations, the effective price per resolved task rises.
• Caching helps only when prefixes repeat. Agents that edit history break the cache.
• Parallel reads trade tokens for time. They cut latency, but reading more pages in parallel raises the token load unless triage limits them.
• Routing needs monitoring. A fixed rule that was right last month can be wrong after a prompt or model change.

Limitations: this is a model, not a measurement. Waste rates, prices, throughputs and quality scores are assumptions. The 4-page naive read and 6-cycle average are typical of research agents but not universal.

Implementation guide.

1. Instrument token usage by stage. Tag every model call with a stage name and task ID; record input, output and cached tokens.
2. Separate reasoning tokens from context tokens. Reasoning is output you can shorten; context is input you can avoid sending.
3. Detect over-reading and duplicate retrieval. Track pages read per page cited, and hash URLs or chunks per session.
4. Route simple steps to cheaper models. Query generation, triage and formatting rarely need the flagship.
5. Compress only after retrieval. Strip boilerplate from pages you chose to read, not blindly beforehand.
6. Add confidence thresholds and fallback. Escalate to the premium model when the cheap model's score is low.
7. Monitor cost, latency and answer quality together. Alert on cost per resolved task, not cost per token.

from collections import defaultdict

def ledger(calls):
    """calls: [{'task','stage','in_tok','out_tok','cached_tok','origin_stage'}]"""
    by_stage = defaultdict(int)
    for c in calls:
        # charge re-sent context to the stage that created it
        by_stage[c.get("origin_stage", c["stage"])] += c["in_tok"] + c["out_tok"]
    total = sum(by_stage.values())
    return {s: round(100 * t / total, 1) for s, t in sorted(by_stage.items(), key=lambda x: -x[1])}

Where Nadir fits.

This is the kind of problem model routing is built for. Instead of treating every step as a flagship-model problem, a team can classify each step's difficulty and send it to the cheapest model that clears a quality floor. Nadir does that with model="auto" and returns per-request routing and cost metadata, so the ledger above can be built from response headers instead of a new tracing stack. Savings vary with your workload, and the stage ledger is how you find out whether your agent looks like this simulation.

Conclusion.

The future of LLM cost optimization is not only cheaper models. It is better orchestration, better routing, better retrieval discipline and better measurement. In our simulation, the stage that mattered was invisible on the invoice. Measure first, then cut.

If you want to see where your own spend goes, estimate your savings, route a slice of your agent traffic, and compare cost, latency and quality per stage against your current baseline.

Sources: [Bai et al., "How Do AI Agents Spend Your Money? Analyzing and Predicting Token Consumption in Agentic Coding Tasks," arXiv:2604.22750](https://arxiv.org/abs/2604.22750). [OpenTelemetry, "Semantic conventions for generative client AI spans"](https://opentelemetry.io/docs/specs/semconv/gen-ai/gen-ai-spans/). Simulation code and parameters are described in the Methodology section; all figures are illustrative.