The $132 Bug Hunt

TL;DR.

A new paper ran four AI security agents against the same real C/C++ bugs and read every trace. The bill made a poor guide to who found the bugs.

• On a 10-task set, the cheapest baseline was the most effective. OpenCode crashed the target on all 10 tasks for $8.58. Cybench crashed 5 of 10 for $49.34.
• On a fresh 20-task set the spread was 30x. OpenCode on a DeepSeek Flash model: 100% success for $4.38. Codex on GPT-5.5: 85% for $132.65.
• 60.4% of all tokens went to two stages: finding and reading code (43.1%) and reasoning about the bug (17.3%). The failures sat in the smaller stage. Reasoning caused 40.3% of failure weight on 17.3% of the tokens.
• Off-the-shelf efficiency add-ons mostly didn't help. Only 24.4% of paired runs both stayed successful and got cheaper. Every add-on raised OpenCode's total cost by 18% to 31%.
• 38 runs never once executed a proof of concept. All 38 failed.

Below: what the paper measured, the five lessons for anyone running agents at scale, and a 70-line Python tutorial that splits your own agent traces by stage, finds the runs burning money blind, and prices a per-stage routing what-if.

Paper figures are from Lu, Zhao, Chen and Wang, ["Where Do the Tokens Go? Understanding and Reducing Costs in LLM Agents for Vulnerability Discovery"](https://arxiv.org/abs/2610.11602), arXiv:2610.11602, submitted October 8, 2026. It is a preprint, every configuration ran once per task, and the failure coding was done by one author. The tutorial output below runs on a synthetic trace and is not Nadir customer data.

Why security agents are the stress test.

Most agent cost writing is about coding assistants: a ticket comes in, a patch goes out. Vulnerability discovery is the same loop with the guardrails removed. The agent gets a repository, a fuzz-driver entry point and a hint about the input format. It gets no bug description, no patch and no reference exploit. It has two hours to produce an input that crashes the vulnerable build.

That makes it an honest worst case. The agent has to search a codebase it has never seen, form a hypothesis, build a payload, run it and read the result. Each step re-reads a growing context. If you run code-review agents, triage agents or anything else that explores a repository, the paper's traces are a preview of your bill.

The benchmark is CyberGym, real vulnerabilities from open-source C/C++ projects. A run counts as a success when the submitted input crashes the vulnerable build. It counts toward recall only when the same input also runs clean on the fixed build, which proves it hit the right bug.

What the paper measured.

Four agents, five configurations each, ten tasks: 200 traces and 776.6 million tokens.

The five configurations were each agent's baseline plus four published efficiency methods:

• AgentDiet compresses the interaction history, using a separate, separately billed compression model.
• Revelio runs lightweight static analysis and hands the agent ranked vulnerability hypotheses.
• PAgent feeds coverage information back to guide proof-of-concept (PoC) generation.
• CodeGraph builds a call and reference graph of the repository with a query interface.

Then the authors hand-coded every trace by stage and every failure by cause. That is the part worth reading.

Lesson 1: the bill does not track the result.

The most effective baseline was also the cheapest, and the most expensive one solved half the tasks. Read the chart top-left to bottom-right: the dots drift the wrong way. More money went to agents that were lost, because a lost agent keeps reading.

That is a standard finding in agent work, and it's why we keep arguing for cost per resolved task instead of cost per token. A token price tells you what the meter charges. It doesn't tell you what a solved problem costs.

Lesson 2: tokens pile up in one stage and failures in another.

The authors split each trace into stages. Two stages dominate spend:

• Code localization and understanding (grep, open, read, re-read): 43.1% of tokens. For EnIGMA it was 60.3%.
• Vulnerability reasoning and trigger design (working out which input reaches the unsafe operation, and under what conditions): 17.3%.

Together that's 60.4%, ranging from 44% for Codex to 71% for EnIGMA. Auxiliary models, the extra models some add-ons call, took another 7.0%.

Then the twist. Across the 67 failed traces, the biggest cause of failure was reasoning and trigger design, at 40.3% of failure weight. Localization came second at 28.4%, and environment problems were third at 16.4%. The stage that eats the most tokens isn't the stage where runs die. Agents spend their budget finding code and fail at thinking about it.

For cost work, that split is the whole game. Reading, searching and navigating is high-volume, low-judgment work, which is the kind cheaper models handle well. Deciding which input reaches a guarded memcpy is low-volume, high-judgment work, the kind that justifies a frontier model. A single-model agent pays frontier prices for both.

Lesson 3: no feedback, no result.

38 of the 200 traces never executed a PoC: 17 from Cybench, 17 from EnIGMA, 4 from Codex and none from OpenCode. All 38 failed. They read and reasoned until the clock or the budget ran out, and never tested a single hypothesis against the target.

That's the cheapest money you will ever save. An agent that has spent a large share of its budget without running its first experiment is unlikely to recover. You don't need a model to detect it. A counter does the job, and the tutorial below includes one.

Lesson 4: efficiency add-ons can cost more than they save.

The paper pairs each add-on run with its baseline on the same task: 160 pairs. Of the 91 pairs where both runs succeeded, only 39 got cheaper with the add-on. That's 24.4% of all pairs. 14 pairs gained a success the baseline missed, 21 lost one, and the net was seven fewer successes.

The cost accounting is where it gets instructive:

• Codex + Revelio: main-model cost fell to 0.61x baseline. Total cost, including the static-analysis preprocessing, rose to 1.15x.
• Codex + AgentDiet: 0.74x over all ten tasks, but 1.10x on the seven tasks both runs solved. The saving came from cheaper failures.
• OpenCode, every add-on: total cost up 18% to 31%, with success flat at 100%. The add-ons only had room to make it worse.

The authors call the first pattern overhead reversal. The main-model meter goes down, the side meters go up, and the dashboard most teams watch only shows the first. We've seen the same thing with prompt compression: a cheaper prompt that breaks the cache, or needs its own model call to produce, can raise the bill. Count every meter or you'll mistake a cost shift for a cost cut.

The most common failure mechanism was an unsuitable signal (59 pairs). The add-on handed the agent hypotheses about a different defect, stub nodes from a call graph, or coverage data that showed where execution went but not why the hypothesis was wrong.

Lesson 5: give the reasoning stage better inputs, not more tokens.

The paper's own fix, AVRI, takes the opposite approach. It doesn't compress anything. It gives the agent a persistent Bidirectional Evidence Trace: a record that tracks how input values flow forward from the harness and which conditions make an operation unsafe working backward, links the two, and keeps the agent's guesses apart from derived facts. The evidence survives context compaction, so the agent doesn't re-derive it after every summary.

On 20 new tasks:

AVRI cut total cost 18.0% for Codex and 23.7% for OpenCode, held success and lifted Codex recall by 10 points. It was the only configuration in the paper that cut cost for both agents without losing success or recall.

Now look at the gap between the agents. On the same 20 tasks, the DeepSeek-powered OpenCode baseline solved every task for $4.38, while Codex on GPT-5.5 solved 17 for $132.65. Per solved task that's about $0.22 against $7.80, a 35x difference. Don't read it as "the cheap model is better": the harnesses differ, each task ran once, and the paper itself names two different models for OpenCode. Read it as a ceiling on how much of a security-agent bill is not buying capability. On these tasks, it was most of it.

Tutorial: audit your agent by stage.

The paper's method works on any agent that logs its model calls. The script below takes one JSON line per model call and answers four questions:

1. Which stage are the tokens and dollars going to?
2. What does a successful run cost, not an average run?
3. Is any side model (compression, judge, summarizer) eating the savings?
4. Which runs never tested anything?

It also prices a what-if: move the high-volume, low-judgment stages to a cheaper model and keep the frontier model for reasoning.

Step 1: log one line per call.

Most agent frameworks expose this already: OpenAI's usage object, Anthropic's usage with cache_read_input_tokens, or your gateway's request log. You need the run ID, model, token counts, the tool the call chose, and whether the run eventually succeeded.

{"run": "t01", "model": "gpt-5.5", "aux": false, "input": 41200, "cached": 30100,
 "output": 900, "tool": "grep", "args": "memcpy src/", "success": null}

Set aux to true for any call made by a helper model rather than the agent itself. That flag is what catches overhead reversal.

Step 2: the audit script.

"""stage_audit.py: where did an agent's tokens go, and did they buy anything?

Input: JSONL, one line per model call:
  {"run": "t01", "model": "gpt-5.5", "aux": false,
   "input": 41200, "cached": 30100, "output": 900,
   "tool": "grep", "args": "memcpy src/", "success": null}
Any line of a run may carry "success": true once the PoC crashes the target.
"""
import json, re, sys
from collections import defaultdict

# $ per 1M tokens: (input, cached input, output). Illustrative list prices
# (DeepSeek off-peak); replace with your own rate card.
PRICES = {
    "gpt-5.5":             (5.00, 0.50, 30.00),
    "deepseek-v4.1-flash": (0.15, 0.003, 0.60),
}

STAGES = [  # first match wins; tune the patterns to your agent's tool names
    ("execute",  re.compile(r"\b(run_poc|submit|gdb|asan)\b|(^|\s)\./")),
    ("build",    re.compile(r"\b(write_file|apply_patch|echo .*>|printf .*>)\b")),
    ("env",      re.compile(r"\b(make|cmake|apt|pip|configure|docker)\b")),
    ("localize", re.compile(r"\b(grep|rg|find|ls|cat|sed -n|read_file|open|search)\b")),
]

def stage(call):
    if call.get("aux"):
        return "aux"
    text = f'{call.get("tool") or ""} {call.get("args") or ""}'
    for name, pat in STAGES:
        if pat.search(text):
            return name
    return "reason"  # no tool call: the model is thinking about what it read

def cost(call, model=None):
    p_in, p_cache, p_out = PRICES[model or call["model"]]
    fresh = call["input"] - call.get("cached", 0)
    return (fresh * p_in + call.get("cached", 0) * p_cache + call["output"] * p_out) / 1e6

runs = defaultdict(list)
for line in open(sys.argv[1]):
    c = json.loads(line)
    runs[c["run"]].append(c)

tok, usd = defaultdict(int), defaultdict(float)
main_usd = total_usd = routed_usd = 0.0
CHEAP, ROUTABLE = "deepseek-v4.1-flash", {"localize", "env"}  # what-if
wins, blind = 0, []
for run, calls in runs.items():
    executed = False
    for c in calls:
        s = stage(c)
        tok[s] += c["input"] + c["output"]
        usd[s] += cost(c)
        total_usd += cost(c)
        main_usd += 0 if c.get("aux") else cost(c)
        routed_usd += cost(c, CHEAP) if s in ROUTABLE else cost(c)
        executed |= s == "execute"
    wins += any(c.get("success") for c in calls)
    if not executed:
        blind.append(run)

T, U = sum(tok.values()), sum(usd.values())
print(f"{'stage':<10}{'tokens':>8}{'cost':>8}")
for s in sorted(tok, key=tok.get, reverse=True):
    print(f"{s:<10}{tok[s]/T:>8.1%}{usd[s]/U:>8.1%}")
print(f"\nruns {len(runs)}, succeeded {wins}, cost per success ${total_usd/max(wins,1):.2f}")
print(f"main-model cost ${main_usd:.2f}, total incl. aux ${total_usd:.2f} "
      f"({total_usd/main_usd:.2f}x)")
print(f"runs that never executed a PoC: {len(blind)} {blind}")
print(f"what-if, {sorted(ROUTABLE)} on {CHEAP}: ${routed_usd:.2f} "
      f"({1 - routed_usd/total_usd:.0%} lower)")

The stage classifier is deliberately dumb: regexes over the tool name and arguments, first match wins. A call with no tool is the model thinking, which is the paper's "reasoning" stage. Tune the patterns to your agent's tool names. Twenty minutes with your own traces beats any generic taxonomy.

Step 3: read the output.

On a synthetic 12-run trace (GPT-5.5 as the agent, a DeepSeek Flash helper summarizing every tenth turn):

stage       tokens    cost
localize     57.0%   62.5%
reason       15.2%   16.2%
aux           9.2%    0.9%
build         7.5%    8.2%
env           6.2%    6.8%
execute       4.9%    5.4%

runs 12, succeeded 8, cost per success $3.51
main-model cost $27.86, total incl. aux $28.11 (1.01x)
runs that never executed a PoC: 3 ['t03', 't08', 't11']
what-if, ['env', 'localize'] on deepseek-v4.1-flash: $9.08 (68% lower)

How to read it:

• `localize` at 57% of tokens and 62.5% of cost is the paper's shape. Searching and reading dominate. That stage is your routing target.
• `aux` at 9.2% of tokens but 0.9% of cost means the helper model is cheap enough to keep. If that last column climbs toward the main-model saving it is supposed to produce, you have overhead reversal. The total incl. aux ratio flags it in one number.
• Cost per success ($3.51) against cost per run ($2.34) is the number to report upward. Failed runs still cost money, so their spend belongs on the runs that succeeded.
• Three runs never executed a PoC. In the paper, every such run failed. Make this a live rule: if a run passes N tokens (or N turns) without a PoC execution or test run, stop it, or escalate it with a fresh plan.
• The 68% what-if prices localization and environment calls at the cheaper model's rate. It's an upper bound, and synthetic, because it assumes the cheaper model reads code as well as the frontier one. Check that on your own tasks before trusting it. The paper's failure data suggests where to be careful: keep the reasoning stage, where 40% of failures start, on the strongest model you can afford.

Step 4: make it a gate.

Run the audit nightly over yesterday's traces and alert on three thresholds:

Five rules for running agents at scale.

1. Price outcomes, not tokens. Cost per solved task is the only number that survives a comparison across models and harnesses. Per-token dashboards made Cybench look like a busy agent, not a lost one.
2. Split the bill by stage. One agent is several workloads with very different difficulty. Searching and reading is volume. Reasoning is judgment. Send each to the model it needs: we went through the routing side of this in Route the Step.
3. Count every meter. Compression models, judges, static analysis and embedding calls all bill. A tool that halves the main-model cost and doubles everything else is a cost shift.
4. Stop blind runs early. No feedback, no result. A counter is enough.
5. Persist what the agent derived. AVRI's saving came from not re-deriving evidence after every compaction. The same idea, applied to reads, is why agents are mostly rereading and why cache hit rate matters so much.

Where Nadir fits.

Rules 1 to 3 are what Nadir does at the gateway. Point your agent's OpenAI-compatible base URL at Nadir and send model="auto". Each call goes to the cheapest model you've configured that clears the quality floor you set per API key, so a grep-and-read turn and a which-input-reaches-the-bug turn don't have to pay the same rate. Every response reports which model answered and what it cost, so the per-call log the tutorial needs builds itself, auxiliary calls included.

If you'd rather measure before changing anything, run a week of agent traffic in Observe mode. It serves exactly the model you asked for and records what routing would have chosen, so the stage-by-stage what-if comes from your own traces instead of our synthetic one.

Start with a free key, or see how this works for coding agents.

Sources: Li Lu, Yanjie Zhao, Hongjie Chen and Haoyu Wang, ["Where Do the Tokens Go? Understanding and Reducing Costs in LLM Agents for Vulnerability Discovery"](https://arxiv.org/abs/2610.11602), arXiv:2610.11602, October 8, 2026, for every success, recall, time, cost, token-share and failure-weight figure. CyberGym benchmark: Wang et al., [arXiv:2506.02548](https://arxiv.org/abs/2506.02548). Per-solved-task figures ($0.22, $7.80) divide the paper's reported cost by tasks solved. DeepSeek V4.1-Flash off-peak list prices from our [October 2026 LLM API pricing comparison](/blog/llm-api-pricing-june-2026-complete-comparison); the GPT-5.5 cached-input rate in the script is illustrative. The tutorial trace and its output are synthetic.