Abstract.
Agent latency is usually reported as a median, but users experience the tail. This post builds a Monte Carlo model of a 12-call sequential agent where each LLM call has a small chance of being a straggler (queueing, a cold replica, a provider brownout). It tests four configurations: premium-only, premium with hedged requests, a two-tier router, and the router with hedging. In this model a 3% per-call straggler rate hits 31% of tasks, routing cuts p95 by 37% and cost by 50%, and hedging adds a further 19% p95 cut for about 6% more spend. Hedging has steep diminishing returns, and routing's escalations create their own tail. Every number is illustrative: it comes from stated assumptions, not production traces.
All data and code in this post are synthetic. Latency distributions, straggler rates and per-call costs are assumptions chosen to be plausible, not measurements of any provider. Treat the shapes as the finding, not the digits.
Why the tail, not the median.
Most LLM cost writing, including much of this blog, asks what a step costs. The question a user feels is different: how long until the agent finishes? An agent is a chain. If a task needs 12 sequential model calls, the task is only as fast as its slowest unlucky call.
That is the old argument from Dean and Barroso's The Tail at Scale (Google, CACM 2013): when a request fans through many components, rare slowness stops being rare. The same arithmetic applies to a loop of LLM calls, with one difference. A fan-out waits for the slowest of N parallel calls, while an agent waits for the sum of N serial ones. Both amplify rare events, and neither shows up in a dashboard that plots median latency per call.
Our earlier work on parallel tool calls covered what parallelism buys inside a step. This post covers the layer above it: what happens across steps, and what routing and hedging do to the distribution.
Research question.
In a sequential LLM agent, how does per-call straggler probability compound into task-level tail latency, and how do model routing and hedged requests change p95 latency and cost?
Methodology.
Task shape. 12 sequential LLM calls per task, 200,000 simulated tasks per configuration. The number is typical of a research or coding agent, not a measurement of one.
Per-call latency. A lognormal body (sigma 0.40) with a median of 4.0 s for a premium model and 1.2 s for a small model, plus a straggler: with probability 3% a call takes 6x its drawn latency. Calls are independent. That assumption is generous, because real provider incidents correlate across calls, which would make tails worse than this model shows.
Per-call cost. Illustrative rates of $0.055 per premium call and $0.011 per small call (a roughly 5x gap, in the range of published tier ratios but not any vendor's rate card).
Configurations.
| Config | Rule |
|---|---|
| Premium only | Every call to the premium model |
| Premium + hedge | If a call has not returned after 8 s, send a duplicate and take the first response |
| Routed | 70% of calls go to the small model; 8% of those fail a verification check and re-run on premium; 40 ms router overhead |
| Routed + hedge | Routed, with the same 8 s hedge on every call |
A hedge is billed at the full call price, which is conservative: if the first request is cancelled, providers may bill less. The routed mix (70/8) is an assumption; real escalation rates depend on the verifier.
Results.
1. Rare stragglers become common tasks.
The probability that a task contains at least one straggler is 1 - (1 - p)^N. At p = 1% and 12 steps that is 11%. At 3% it is 31%. At 5% it is 46%. A call-level SLO that looks healthy (97% of calls are normal) becomes a task-level problem for nearly a third of runs. Longer agents make it worse: at 30 steps and p = 3%, 60% of tasks hit at least one.
2. Routing moves the whole curve; hedging trims the right edge.
| Config | p50 (s) | p95 (s) | p99 (s) | Cost/task |
|---|---|---|---|---|
| Premium only | 55.1 | 91.7 | 113.7 | $0.66 |
| Premium + hedge @8s | 54.1 | 69.0 | 76.9 | $0.71 |
| Routed | 31.9 | 57.7 | 76.9 | $0.33 |
| Routed + hedge @8s | 31.6 | 47.0 | 54.9 | $0.35 |
Hedging barely moves the median (55.1 s to 54.1 s) but cuts p95 by 25%, which is its whole purpose. Routing moves everything because most calls now hit a faster, cheaper model. Note the routed p99 (76.9 s) is much closer to its p50 than the premium-only gap, yet it still carries a visible shoulder: the escalations. A task that runs a small call, fails verification, then runs a premium call pays for both in series.
3. Hedging has a steep knee.
Sweeping the hedge delay on the premium-only agent: hedging at 14 s buys a 17% p95 cut for 2.8% extra spend. Dropping the delay to 8 s adds a few more points of p95 for 7% extra. At 4 s, p95 improves only slightly more while spend rises 51%, because by then you are duplicating ordinary calls, not stragglers. The right delay sits near the p90-p95 of the single-call distribution, not its median.
4. Routing and hedging attack different things.
In this model, routing is the only lever that lowers cost and latency together, because it changes which model does the work. Hedging is a pure latency purchase: +7% cost for -25% p95 on the premium-only agent, +6% cost for -19% on the routed one. The combination is lowest on both axes among the four, but the hedge premium is smaller in absolute terms on the routed agent because its base calls are cheaper.
Discussion.
Caveats first. This is a model, not a measurement. Independent stragglers are optimistic. A 70% small-model share is only available if your tasks have that much simple work, and routing quality is a separate question this simulation does not touch: it assumes the small model's answers are acceptable when they pass verification.
Hedging is not free, and it is not safe by default. A duplicated call is a duplicated bill. For calls with side effects (tool calls that write, send or charge) a hedge can execute an action twice. Hedge idempotent model calls, never non-idempotent tools.
Escalation creates its own tail. Cheap-first routing lowers the median but adds a serial small-then-premium path for the calls that fail checks. If your verifier is slow, that path is the new p99. Cache behavior matters too: an escalation to a model with a cold prefix cache adds time-to-first-token on top.
Step caps are a latency control. Because tail exposure grows with N, capping steps bounds both runaway cost and the probability of meeting a straggler.
Implementation guide.
Step 1: Record per-call latency with percentiles, per model and per step type. Median alone hides the tail.
Step 2: Compute your real straggler rate. Define a straggler as a call over, say, 3x its model's median. That one number drives the compounding in Chart 1.
Step 3: Roll up to the task level. Report p95 and p99 per task, not per call. Compare against 1 - (1 - p)^N to see whether stragglers are independent or clustered.
Step 4: Set per-call timeouts before hedging. A timeout with fallback to another provider or model is the cheap version of a hedge.
Step 5: Hedge only idempotent model calls, at roughly your p90-p95 call latency. Measure the duplicate-spend percentage and stop when the knee in Chart 3 appears.
Step 6: Route the easy steps to a faster model, with a verification check and an escalation budget. Log how many tasks take the escalation path and how long it adds.
Step 7: Track cost, p95 latency and answer quality on one dashboard. Changing any one of them without the others is how a latency win becomes a quality regression.
import numpy as np
def straggler_rate(latencies_s, model_median_s, k=3.0):
lat = np.asarray(latencies_s)
return float((lat > k * model_median_s).mean())
def p_task_hits_straggler(p, n_steps):
return 1 - (1 - p) ** n_steps
# e.g. p = 0.03, n_steps = 12 -> 0.306The product angle.
Steps 1, 3 and 7 are an instrumentation problem before they are an optimization problem. A team that only sees the cost per call cannot tell whether its p95 comes from a slow model, a hot spot, or a chain of escalations.
This is the kind of problem a routing and observability layer is built for. Instead of treating every step as a premium-model problem, a router can classify each step, send simple ones to a faster, cheaper model, escalate when a check fails, and log latency, cost and outcome per request, so the distribution above is something you can compute from your own traffic. Nadir is built around that loop.
Conclusion.
The future of LLM cost optimization is not just cheaper models. It is better orchestration, better routing, better retrieval discipline, and better measurement, and for agents, measurement has to include the tail. A 3% slow-call rate is invisible in a per-call median and shows up in a third of finished tasks. Route what is simple, hedge what is idempotent, cap what can loop, and watch p95 next to cost.
[Benchmark your own agent's token spend and latency distribution →](/auth?mode=signup)
All figures in this post are illustrative, from a seeded Monte Carlo simulation (200,000 tasks per configuration, 12 sequential calls, lognormal latency with a 3% 6x straggler component, illustrative per-call costs of $0.055 and $0.011). They are not measurements of any provider or production system, and results will differ with your latency distributions, escalation rate and hedging policy. Source: Dean, J. and Barroso, L. A., [The Tail at Scale](https://research.google/pubs/the-tail-at-scale/), Communications of the ACM, 2013.