Abstract Every team building an AI agent in 2026 has been told, correctly, that evals are the new unit tests: nothing ships without automated proof it meets a correctness spec, and the discipline has grown fast enough that "evals engineer" is now a real job title. Source: FutureAGI, "What is Evals Engineering? The Discipline Behind Production LLMs in 2026". What that framing leaves out is that an eval suite is itself an LLM workload, usually a bigger one than the system it's grading, and it inherits every cost problem this blog tracks in production traffic: no caching, no routing, frontier models called by default, and nobody watching the bill because the spend hides inside an internal tooling budget instead of a customer-facing product. Princeton's Holistic Agent Leaderboard puts a real number on this: $40,000 to run 21,730 agent rollouts once, across 9 models and 9 benchmarks, and roughly $320,000 to make that same benchmark statistically reliable. Source: Kapoor et al., "Holistic Agent Leaderboard: The Missing Infrastructure for AI Agent Evaluation," arXiv:2510.11977, ICLR 2026. This post uses that paper and the wider literature it sits inside to show where eval spend actually goes, why higher cost doesn't reliably buy higher accuracy, and why the fix, cascade the judge, cache the reruns, route to a distilled model first, is the same architecture this blog has already argued for on the production side. The benchmark you didn't budget for Ask a team what their AI spend looks like and they'll describe the production bill: tokens per request, requests per day, model tier. Ask what their CI pipeline spends re-running the eval suite on every pull request, or what last week's model-upgrade regression test cost, and most teams don't know, because nobody logs it as its own line item. Langfuse's own community has flagged this directly: their platform didn't track LLM-as-judge token usage or cost at all until October 2025, and the standard workaround before that was a separate API key so the eval spend could at least be found in a different dashboard. Source: Langfuse, GitHub Discussion #7166, "Cost and token usage for LLM-as-a-Judge evaluations". That's not a tooling gap that stays small. As agentic systems get harder to grade with a regex or an exact-match check, teams increasingly reach for an LLM to read the transcript and score it, LLM-as-judge, and that judge call is a second inference request stacked on every graded interaction, run not once but every time the eval suite executes: on every commit, every nightly regression run, every model swap, every prompt tweak somebody wants to A/B before shipping. The math that made routing and caching the right answer for production traffic, most calls are simple, most content repeats, frontier tier isn't required for most of it, applies to this second workload with just as much force. It just doesn't show up on anyone's dashboard yet. Research question Where does eval and benchmark spend actually go, does more of it produce a more accurate result, and does the cascade-and-cache architecture that cuts cost on the primary model call apply with equal force to the pipeline that grades it? Methodology This analysis combines two kinds of source, kept clearly separate: Published, primary research on the real cost of running agent benchmarks: Princeton's Holistic Agent Leaderboard paper, accepted to ICLR 2026, which ran a standardized harness across 9 benchmarks and 9 models and logged the actual dollar cost. Source: Kapoor et al., arXiv:2510.11977. An independent reproduction of a subset of that work arrived within a few thousand dollars of the original figure on a completely separate run. Source: EvalEval Coalition, "AI evals are becoming the new compute bottleneck," April 29, 2026. Published vendor benchmarks for judge-model cost and latency, specifically Galileo's Luna-2 evaluator, a 2-billion-parameter model published against frontier judges on cost, latency, and agreement with human labels. Source: Galileo, Luna-2 model benchmarks. Where a figure is a direct measurement from one of these sources, it's marked as such. Nothing in this piece is our own production data; this is a review of the current research and vendor literature on eval cost, the same way earlier posts on this blog have reviewed FrugalGPT, RouteLLM, and LLMRouterBench. Results: a real benchmark, priced The clearest anchor here is Princeton's HAL project, built specifically because agent benchmarking had no shared, cost-controlled infrastructure: every lab was reporting accuracy numbers from harnesses nobody else could reproduce, at a cost nobody else could see. Running the same standardized harness across 9 benchmarks and 9 models, 21,730 rollouts, cost $40,000 once. Source: Kapoor et al., arXiv:2510.11977. By April 2026 the leaderboard had grown to 26,597 rollouts, and an independent researcher's reproduction of a comparable slice landed at $46,000 across 242 runs, close enough to the original to confirm the number isn't an outlier. Source: EvalEval Coalition, April 2026. A single run isn't a reliable number, though; LLM outputs are stochastic enough that any one rollout can land above or below a model's true accuracy by chance. Getting a statistically defensible result means rerunning each task multiple times, and the standard reliability protocol HAL's own team recommends is 8 reruns per task. Applied to the same 21,730-rollout benchmark, that turns a $40,000 single pass into roughly $320,000, an 8x multiplier that most teams simply don't pay, which is a separate problem from the one this post is about, but worth naming: a lot of published agent-benchmark numbers are one noisy sample, not a mean. The eval tax, in two charts: making Princeton's HAL agent benchmark statistically reliable turns a $40,000 single pass into $320,000, an 8x multiplier from running 8 reruns per task instead of one. On Online Mind2Web, Browser-Use with Claude Sonnet 4 cost $1,577 for 40% accuracy, while SeeAct with GPT-5 Medium cost $171 for 42% accuracy, a 9x cost difference for a 2 point accuracy difference in the other model's favor. Cost doesn't just scale unpredictably across the whole HAL suite, four orders of magnitude between the cheapest and most expensive benchmark it runs, three orders of magnitude within some individual benchmarks depending on scaffold choice. Source: EvalEval Coalition, April 2026. It also doesn't correlate with accuracy the way a naive "spend more, score better" assumption would predict. On the Online Mind2Web benchmark, Browser-Use paired with Claude Sonnet 4 cost $1,577 to reach 40% accuracy, while SeeAct paired with GPT-5 Medium cost $171 and reached 42%, a nine-fold cost difference for a result that was, if anything, slightly worse for the money. Source: HAL / EvalEval Coalition, April 2026. A separate study across six state-of-the-art agents on 300 enterprise tasks, Agentic CLEAR, found the same shape at a broader scale: accuracy-optimal configurations cost 4.4 to 10.8 times more than the Pareto-efficient alternative that reached comparable accuracy for less. Source: "Agentic CLEAR: Automating Multi-Level Evaluation of LLM Agents," arXiv:2605.22608. And a separate sweep across agent configurations on identical tasks found a 33x cost spread driven almost entirely by scaffold choice, not model choice. Source: EvalEval Coalition, April 2026. Read together, these aren't isolated anomalies. They're the same finding this blog has made about production routing applied to the harness that grades it: throwing the most expensive available option at every task is not the same thing as buying quality, and the gap between the two is often large enough to fund the entire routing layer that would have closed it. Results: the judge call is its own bill, and it scales the same way The other place eval spend concentrates is the judge itself, the model reading a transcript or an output and scoring it, run on every graded item, every time. A production team using LLM-as-judge for continuous quality monitoring runs this call constantly: on sampled live traffic, on every regression suite, on every offline eval sweep before a release. Purpose-built judge models exist precisely because a frontier general-purpose model is overkill for most of this scoring. Galileo's Luna-2, a 2-billion-parameter model fine-tuned specifically for hallucination, context adherence, and tool-call correctness, publishes benchmarks directly against frontier judges: $0.12 per million tokens and 152ms average latency, against GPT-5.4's $5.00 per million tokens and 3,200ms, a roughly 41x cost difference and a roughly 21x latency difference, while scoring 0.95 agreement with human labels against GPT-5.4's 0.94. Source: Galileo, Luna-2 published benchmarks. That's not a company claiming its cheap model is "almost as good." It's a smaller, purpose-trained model matching a general-purpose frontier model on its own scoring task, at a fraction of the price and a fraction of the wait. What a judge call actually costs, per million tokens: GPT-5.4 as a general-purpose frontier judge runs $5.00 per million tokens at 3,200ms latency with 0.94 agreement against human labels. Azure Content Safety runs $1.52 per million tokens at 312ms. GPT-5.4 mini runs $0.15 per million tokens. Galileo's Luna-2, a 2-billion-parameter model fine-tuned specifically as a judge, runs $0.12 per million tokens, about 41 times cheaper than the frontier judge, at 152ms latency and 0.95 agreement, matching or beating the frontier model it's compared against. The applied guidance in the LLM-as-judge literature has converged on a similar rule of thumb to what production routing teams already use for the primary model call: distilled, purpose-built judges cut cost 10 to 50x against a frontier general-purpose judge, judge spend should stay under roughly 10-15% of the production LLM cost it's monitoring, and if judge spend crosses roughly 25% of that budget the right move is a lower sample rate, not a bigger judge budget. Source: FutureAGI, "LLM-as-Judge Best Practices in 2026: Calibration, Bias, and Cost". Anthropic's own published guidance for building agent evals points at the same discipline from a different angle: use deterministic, code-based graders wherever the task allows it, since they're both faster and free of judge-model noise, and reserve LLM graders, and human review of LLM graders, for the subjective judgment calls that a script genuinely can't make. Source: Anthropic, "Demystifying evals for AI agents". Results: you can shrink the benchmark itself, not just the judge A second, complementary line of research attacks the problem from the other side: instead of making each evaluation call cheaper, run fewer of them without losing the signal. Static benchmark compression is furthest along here. tinyBenchmarks reduced MMLU from roughly 14,000 items to 100 while holding ranking error to around 2%, and Anchor Points showed that ranking 87 model-and-prompt combinations reliably needed as few as 1 to 30 carefully chosen examples rather than the full set. Source: EvalEval Coalition, April 2026. Flash-HELM applies the same idea as a coarse-to-fine funnel, spending compute on a cheap broad pass across every candidate and reserving the expensive high-resolution evaluation for the shortlist that pass produces, reporting 100 to 200x compute reduction while preserving the final rankings. Source: EvalEval Coalition, April 2026. Agent benchmarks compress less cleanly, since a multi-step trajectory doesn't reduce to a single item the way a multiple-choice question does, but mid-difficulty filtering, dropping the tasks every model already passes and the tasks every model already fails, still recovers a real 2 to 3.5x reduction while preserving how models rank against each other. Source: EvalEval Coalition, April 2026. None of this is a new idea dressed up for AI evals specifically. It's the same principle behind prompt compression on the production side: most of the content you're paying to process isn't carrying the signal you actually need, and the fix is figuring out which part is. Discussion: the eval pipeline needed the same fix production traffic did A judge call is a model call, and it should be routed like one. The same escalation logic Anthropic applies to its own safety classifiers, cheap check first, expensive check only on the fraction that's genuinely ambiguous, applies directly to a grading pipeline: a distilled judge scores the overwhelming majority of items correctly on its own, and only the low-confidence or disagreement cases need to escalate to a frontier model. Running every graded item through a frontier judge by default is the same predict-and-hope waste this blog has flagged in naive model routing, just applied to the call that checks the work instead of the call that does it. Reruns and repeated content are cacheable, and most eval pipelines don't cache them. A regression suite re-run on every commit sends largely the same prompts and largely the same reference transcripts through the judge every single time a developer pushes a small, unrelated change. The same prefix-caching discipline this blog has covered for production system prompts applies just as directly to a CI eval run: static reference answers, rubric text, and unchanged transcript segments are exactly the kind of stable prefix caching exists for. More expensive doesn't mean more accurate, on the harness side either. The Mind2Web and Agentic CLEAR findings above are a direct echo of what routing research already found on the model-selection side: Berkeley's RouteLLM work showed 74% of GPT-4-tier calls didn't need GPT-4-tier quality, and the same gap between spend and outcome shows up when the thing being evaluated is the benchmark harness rather than the production request. This is a governance gap before it's a technical one. The single biggest reason eval spend goes unmanaged isn't that the fixes are unknown, cascading judges and caching reruns are both well-documented. It's that most teams don't instrument the eval pipeline as its own cost center at all, the same blind spot FinOps research has already documented for AI spend broadly. You can't route or cache a bill you've never separated from the rest of your token spend. Implementation guide Give eval spend its own line item. Break out judge-model tokens, regression-suite reruns, and offline benchmark sweeps from the production bill they're currently hiding inside. You can't fix what isn't visible, and most observability stacks didn't even log this by default until late 2025. Start small and let it grow with the signal, not the budget. Anthropic's own guidance is to begin with 20-50 real failure cases rather than a sprawling suite; early in development, each fix has a large, obvious effect, so a small sample is enough to see it. Source: Anthropic, "Demystifying evals for AI agents". Prefer a deterministic grader wherever the task allows it. Exact-match, schema validation, and code-execution checks are free of judge-model cost, latency, and drift. Reserve the LLM judge for the subjective calls a script genuinely can't make. Route the judge call itself. Score with a distilled, purpose-built judge model first, and escalate to a frontier judge only on low-confidence or disagreement cases. This is a model-routing cascade aimed at your eval pipeline instead of your production traffic, and the tooling is the same. Cache what repeats. Reference answers, rubric instructions, and unchanged transcript segments across CI reruns are a stable prefix; cache them instead of re-scoring them from scratch on every commit. Set a budget ratio and act on it. A commonly used floor is judge spend under roughly 10-15% of the production cost it monitors; crossing roughly 25% is a signal to lower the sample rate, not to keep the judge budget open-ended. Source: FutureAGI, "LLM-as-Judge Best Practices in 2026". Compress the benchmark before you compress the judge. If a suite has grown large through accumulation rather than design, a mid-difficulty filter or an anchor-point subset can cut the item count several-fold before touching per-item cost at all. Report cost per statistically reliable result, not cost per run. A single noisy rollout is not a benchmark score. If reliability requires reruns, budget for the reruns up front instead of discovering the real number later. The eval pipeline's judge call is a normal chat completions request. Point it at Nadir the same way you would a production call: a cheap, distilled judge handles the item first, and only low-confidence or disagreement cases escalate to a frontier-tier judge. import openai client = openai.OpenAI( base_url="https://api.getnadir.com/v1", # was: https://api.openai.com/v1 api_key="YOUR_NADIR_KEY", ) verdict = client.chat.completions.create( model="auto", # Nadir routes per item; the eval suite doesn't have to guess messages=[ {"role": "system", "content": JUDGE_RUBRIC}, # stable across reruns — cached {"role": "user", "content": f"Transcript:\n{transcript}\n\nReference:\n{reference}"}, ], max_tokens=200, # a verdict and a rationale, not an essay ) Conclusion Nobody set out to build an unmanaged second AI bill next to their production one. It happened the same way the production bill got out of hand in the first place: a reasonable default, call the frontier model, check every item, spent without a second thought because the alternative wasn't obvious yet. Princeton's HAL project put a real number on what that default costs at benchmark scale, $40,000 for one honest pass at 21,730 rollouts, $320,000 to make it a number you can trust, and the wider literature it sits inside shows the same pattern this blog has tracked in production traffic for months: spend doesn't track with quality, a purpose-built model can match a frontier one on the narrow task of scoring, and most of what a suite reruns hasn't actually changed since the last run. None of that is an argument against evals. Eval-driven development is the right instinct, and a suite you can't trust is worse than no suite at all. It's an argument for treating the harness that grades your agent with the same discipline you'd apply to the agent itself: route the judge call to the model that can actually make the call, cache the reference content that hasn't changed, and know what the number costs before it shows up as a surprise in next quarter's tooling budget. Nadir already routes and caches the production call; pointing the same base URL at your judge pipeline is the same fix, aimed at the bill nobody's been watching. Sources: Kapoor et al., "Holistic Agent Leaderboard: The Missing Infrastructure for AI Agent Evaluation," arXiv:2510.11977, ICLR 2026. EvalEval Coalition, "AI evals are becoming the new compute bottleneck," April 29, 2026. Langfuse, GitHub Discussion #7166, "Cost and token usage for LLM-as-a-Judge evaluations". Galileo, Luna-2 published benchmarks. FutureAGI, "LLM-as-Judge Best Practices in 2026: Calibration, Bias, and Cost". FutureAGI, "What is Evals Engineering? The Discipline Behind Production LLMs in 2026". Anthropic, "Demystifying evals for AI agents". "Agentic CLEAR: Automating Multi-Level Evaluation of LLM Agents," arXiv:2605.22608.