One Call, Sixteen Answers

Abstract.

Most bulk LLM jobs pay for the same instructions over and over. A ticket classifier with a 1,800-token system prompt and few-shot examples, labeling a 150-token ticket, spends 92% of its input bill on the part that never changes. Batch prompting is the old, unglamorous fix: put 16 tickets in one call, ask for 16 labels back, and pay for the instructions once. A May 2026 paper, RoBatch (Xu, Zhao, and Xie, arXiv:2605.28268), shows why teams should treat it as a routing problem rather than a formatting trick. Batch size and model choice interact, the joint problem is NP-hard, and accuracy falls off a cliff at different batch sizes for different tasks: past 16 items on AGNews classification, past 8 on GSM8K math. This post models a one-million-item job at current list prices, shows that batching is worth about 80% on its own but only about 30% once prompt caching is on, and walks through a working implementation: bucket each item by difficulty, batch within each bucket at a size that bucket can take, validate every answer by id, and send failures back one at a time. Stacked with caching, routing, and a provider Batch API, the modeled bill falls from $4,100 to $237 per million items.

All costs and charts in this post are illustrative, modeled from public list prices and the cited papers, not measured production traces. Not derived from proprietary customer data. Sources cited throughout.

The research question.

There are two different things people call "batching" in LLM work, and they get confused constantly.

We covered the Batch API discount already. This post is about the second one, which changes what the model actually reads, and therefore changes what it gets right.

Batch prompting was formalized in 2023 by Cheng, Kasai, and Yu ("Batch Prompting: Efficient Inference with Large Language Model APIs," arXiv:2301.08721, EMNLP 2023), who reported token and time costs falling by up to 5x with six samples per batch, with downstream accuracy held on commonsense QA, arithmetic, and NLI. Their observation was that cost falls "almost inverse linearly" with the number of samples per batch, because the few-shot exemplars dominate the prompt. Three years later, that observation is truer than ever. System prompts have grown, often to the point of bloat, and bulk jobs (classification, extraction, moderation, tagging, eval grading) are where a growing share of enterprise token spend now goes.

RoBatch adds the piece that was missing: which model should a batch go to, and how big should that batch be for that model? The authors formulate this as the Route with Batching Problem, prove it NP-hard, and solve it with a batch-aware utility estimate plus greedy scheduling along the cost-utility Pareto frontier. Across six benchmarks and two model families (Qwen3 4B, 14B, 32B and Gemma3 4B, 12B, 27B), joint route-and-batch beat both router-only and batch-only baselines in their ablations.

The research question for this post: on a realistic bulk job at today's prices, how much does batch prompting actually save, how does that change once the other standard levers are on, and how do you pick batch sizes without losing the accuracy you routed for?

Methodology.

An analytical cost model, not a benchmark. The accuracy boundaries come from the RoBatch paper; every price is a September 2026 list price; everything else is a stated assumption.

The per-item input cost of a batch of size b is (prefix ÷ b) + item. That's the whole mechanism. When the prefix is 12x larger than the item, as here, the first few doublings of b remove most of the bill, and every doubling after that removes less.

Finding 1: batching is an 80% lever, until you cache.

On a cold prefix, batch prompting is dramatic. Haiku 4.5 at one item per call costs $2,050 per million items; at 16 items per call it's $402.50, an 80.4% cut, with no model change and no discount program.

Turn on prompt caching first, though, and the same move is worth 30%. Caching already bills the 1,800-token prefix at a tenth of the input rate, so there's much less left for batching to amortize. This is the single most useful thing to know before you rewrite a pipeline: if your bulk job already hits a warm prompt cache, batch prompting is a 30% lever, not an 80% one. It still stacks, and 30% of a large bill is real money, but it's not the headline some write-ups imply.

The reverse is also true, and it's why batching is still worth doing. Caches are fragile in exactly the workloads where batching fits. A nightly job that sends a request every few seconds can fall out of a five-minute cache window, a queue that fans out across many workers can miss the cache on every cold start, and cache hits inside provider batch jobs are best-effort. We wrote about what cache writes cost when the hit rate slips. Batch prompting doesn't depend on timing. The saving is baked into the request itself.

Finding 2: the curve flattens exactly where accuracy starts to fall.

The cost curve is a hyperbola, and hyperbolas have a knee. Between 1 and 8 items per call, the uncached bill falls 75%. Between 16 and 64 it falls another 21%. With a warm cache, 16 to 64 buys you about 3%.

Now lay RoBatch's accuracy results over it. In their experiments:

• AGNews (news topic classification) held its accuracy up to a batch size of 16, then degraded.
• GSM8K (grade-school math, multi-step reasoning) held only up to 8.
• Qwen3-4B, the smallest model tested, dropped sharply above a batch size of 50.
• The authors' explanation: as the batch grows, "the excessive complexity of the concatenated prompt overwhelms the model's reasoning capability."

The same paper quantifies the other side. At a batch size of 1, the shared system prompt was 59.5% of total cost on AGNews and 90.1% on GSM8K. At 16 items on AGNews its share fell to 8.4%; at 8 items on GSM8K, to 53.2%.

Put the two together and the practical rule falls out: the batch sizes that hurt accuracy are the ones that barely save money. The expensive mistake isn't batching too little. It's pushing a reasoning task to 32 or 64 items per call to save the last few percent, and quietly giving back the accuracy you were paying a better model to get.

Finding 3: batch size is a routing decision.

Here is where the RoBatch framing earns its keep. The naive way to batch is to pick one model, pick one batch size, and push the whole queue through. That forces a bad compromise: a batch size safe for your hardest items is wasteful for your easiest ones, and a model strong enough for the hardest items is overkill for most of the queue.

All-Haiku at 16 per call is the cheapest line on the table, and it's the one that fails in the way that's hardest to notice: the easy 70% of the queue looks fine in a spot check, and the 5% of genuinely hard tickets get confidently mislabeled. Bucketing first costs $173 more per million items than that, and spends it on exactly the items that need it. It's the same argument as routing each prompt to the cheapest model that can handle it, applied one level up: the unit you route is now a batch, and the batch inherits the difficulty of its hardest member.

That last point is the practical reason to bucket before you batch rather than after. If you batch first and route the batch, one hard item in sixteen drags the whole call to an expensive model. Group by difficulty first and a batch is homogeneous by construction.

Tutorial: bucket, batch, validate, fall back.

Four steps. The code uses the OpenAI Python SDK, so it works with any OpenAI compatible endpoint, including Nadir's.

Step 1: bucket every item.

import httpx, os

NADIR = "https://api.getnadir.com/v1"
HEADERS = {"X-API-Key": os.environ["NADIR_API_KEY"]}

def bucket(item_text: str) -> str:
    r = httpx.post(f"{NADIR}/bucket", headers=HEADERS,
                   json={"prompt": item_text, "source": "direct"}, timeout=10)
    r.raise_for_status()
    return r.json()["routing_tier"]   # "simple" | "medium" | "complex"

/v1/bucket is the bare complexity classifier: it returns a tier and spends no LLM tokens. If you'd rather use your own classifier, a heuristic, or a label you already have, that's fine. What matters is that every item has a difficulty before it's grouped.

Step 2: pick a model and a batch size per lane.

LANES = {
    "simple":  {"model": "claude-haiku-4-5", "batch": 16},
    "medium":  {"model": "claude-sonnet-5",  "batch": 8},
    "complex": {"model": "claude-opus-5",    "batch": 1},
}

def plan(items):
    groups = {tier: [] for tier in LANES}
    for item in items:
        groups[bucket(item["text"])].append(item)
    for tier, members in groups.items():
        size = LANES[tier]["batch"]
        for i in range(0, len(members), size):
            yield tier, members[i:i + size]

Start conservative. The RoBatch boundaries (16 for classification, 8 for reasoning) are a reasonable first guess, not a law. Your eval set decides the real numbers, per lane.

Step 3: ask for answers keyed by id, and check every one.

import json
from openai import OpenAI

client = OpenAI(base_url=NADIR, api_key=os.environ["NADIR_API_KEY"])
SYSTEM = open("ticket_labeler_prompt.txt").read()   # the 1,800-token prefix, unchanged

def run_batch(model, members):
    body = "\n".join(f'<item id="{m["id"]}">{m["text"]}</item>' for m in members)
    resp = client.chat.completions.create(
        model=model,
        messages=[
            {"role": "system", "content": SYSTEM},
            {"role": "user", "content":
                f"Label each item independently. Return JSON: "
                f'{{"results": [{{"id": ..., "intent": ..., "priority": ...}}]}}, '
                f"one entry per item, same ids.\n{body}"},
        ],
        response_format={"type": "json_object"},
    )
    try:
        rows = json.loads(resp.choices[0].message.content)["results"]
    except (json.JSONDecodeError, KeyError, TypeError):
        return {}, [m["id"] for m in members]
    want = {m["id"] for m in members}
    got = {r["id"]: r for r in rows if r.get("id") in want and valid(r)}
    return got, sorted(want - got.keys())

Three details carry most of the reliability:

• Put the shared prefix first and keep it byte-identical. The batch goes after it, so the prefix stays cacheable and the two levers stack. Anything per-batch, even a timestamp, belongs after the prefix.
• Key every answer by an id you chose. Never trust position. Models skip, merge, and reorder items in long batches, and a positional parser will silently assign ticket 9's label to ticket 10.
• Validate each row on its own (valid() checks the label is in your taxonomy). A batch isn't pass or fail; each item is.

Step 4: send the misses back one at a time.

def process(items):
    results = {}
    for tier, members in plan(items):
        got, missing = run_batch(LANES[tier]["model"], members)
        results.update(got)
        for item_id in missing:                        # unbatch on failure
            item = next(m for m in members if m["id"] == item_id)
            single, _ = run_batch(LANES[tier]["model"], [item])
            results.update(single)
    return results

Retry failures as singles, not as a fresh batch. A malformed batch usually means the batch was too crowded for that model, so re-sending the same crowd tends to reproduce the failure. Track the miss rate per lane. If it climbs above a few percent, the lane's batch size is too big, and every retry is quietly eating the saving; we've written about how retries erode small-model savings in more detail.

Once this runs, the Batch API is one more flag on the same prompts. Stacking both in this model halves $474.63 to $237.31.

When not to batch.

• Anything a user is waiting on. A batch returns when its last item finishes. Batch prompting is for queues, not chat.
• Items from different customers, if your contract or DPA cares. Mixing two tenants' records in one prompt puts one customer's data in the context of another's answer. Batch within a tenant.
• Agentic and tool-using work. A tool call answers one item's question with the whole batch in context. Keep agent loops unbatched.
• Long items. The saving comes from the prefix being much larger than the item. At a 1,800-token prefix and a 2,000-token item, 16 items per call cuts input cost by 44% instead of the 87% it cuts here, and the context grows to 34,000 tokens.
• Outputs that are long or free-form. Batching doesn't reduce output tokens; our model shows it slightly increases them (28 vs. 20 per item) for the id and framing. For summaries or drafts, the output dominates and the prefix saving is a rounding error. Look at trimming the output instead.

How to measure it on your own queue.

Before touching production, run the eval that decides batch size:

1. Take 500 labeled items per difficulty lane.
2. Run each lane at batch sizes 1, 4, 8, 16, and 32 on the lane's model.
3. Plot accuracy and cost per item against batch size, per lane.
4. Also plot accuracy by position within the batch. If slot 14 of 16 is consistently worse than slot 2, the batch is too big even if the average looks fine.
5. Pick the largest size whose accuracy is within your noise floor of batch size 1.

That's a few thousand calls, a small fraction of one day's bill on a million-item queue, and it replaces a guess with a number.

Where Nadir fits.

Batch prompting is something you do in your own pipeline; nothing about it requires a gateway. What it does require is a difficulty signal per item, and that's the part Nadir provides. POST /v1/bucket classifies each item without spending provider tokens, and if you pass a ladder mapping tiers to your own models, the response carries selected_model so the lane assignment isn't code you maintain. Through the OpenAI compatible gateway, every batched call then reports its actual cost per request, so you can check the lane-by-lane arithmetic in this post against your own traffic instead of our assumptions.

If you only want the decision, not the proxy, the bucket endpoint works on its own and your existing client keeps making the calls. Start with a free key, bucket one day of your queue, and see how much of it really needed the model you're sending all of it to.

Conclusion.

Batch prompting is a three-year-old idea that got more valuable as system prompts grew, and more dangerous as teams started pushing it without measuring. The numbers say three things. Turn on prompt caching first, because it captures most of what batching would. Batch anyway, because caches miss and batching doesn't. And size each batch to the difficulty of what's in it, because the batch sizes that lose accuracy are the ones that barely save money. Bucket, batch, validate by id, unbatch on failure. On this model that takes a million-item job from $4,100 to $237, and none of the saving comes from asking a small model to do a big model's work.

Costs in this post are illustrative, modeled for this tutorial from public list prices, and are not derived from customer data. Sources: [Xu, Zhao, and Xie, "Towards Cost-effective LLMs Routing with Batch Prompting," arXiv:2605.28268, May 2026](https://arxiv.org/abs/2605.28268). [Cheng, Kasai, and Yu, "Batch Prompting: Efficient Inference with Large Language Model APIs," arXiv:2301.08721, EMNLP 2023](https://arxiv.org/abs/2301.08721). [Anthropic, Pricing](https://platform.claude.com/docs/en/about-claude/pricing). [Anthropic, Prompt caching](https://docs.anthropic.com/en/docs/build-with-claude/prompt-caching). [Anthropic, Message Batches API](https://docs.anthropic.com/en/docs/build-with-claude/batch-processing).