Right Answer, Wrong Shape

Abstract.

A small model returns {"total": 42,} with a trailing comma. Your validator rejects it, your cascade marks the request as failed, and a flagship model is paid to produce {"total": 42}. The answer was right the first time. Only its shape was wrong. Two 2026 papers put numbers on how often this happens. Galeone et al. (arXiv:2605.02363, May 2026) found 7-9B models reaching up to 85% task accuracy on GSM8K with 0% output accuracy under a naive prompt: the math was right and the JSON was unusable, every time. Reddy et al. (arXiv:2603.03305) showed that constraining a 1B model's output directly cost it accuracy, and that letting it draft freely first and then constraining lifted structured accuracy from 15.2% to 39.0%. A third paper, Lee (arXiv:2608.13959, August 2026), adds the warning that keeps this honest: constrained decoding "gives back readable output, not judgment." This post separates format failures from content failures, models what each one costs in a routed pipeline, and walks through a triage step in Python that fixes the cheap failures cheaply and escalates only the ones a bigger model can actually fix. On a million-extraction workload at September 2026 list prices, that moves the bill from $3,900 to $2,941 against a cascade that escalates every failure, and from $9,750 against sending everything to the flagship.

All costs and charts in this post are illustrative, modeled from public list prices and the cited papers, not measured production traces. Not derived from proprietary customer data. Sources cited throughout.

The research question.

Every cascade has the same shape. Try the cheap model, check the answer, escalate if the check fails. We've written about why the check matters and what blind retries cost. This post is about something narrower: what the check is actually detecting.

For any workload that needs structured output (extraction, classification with metadata, tool arguments, form filling), the first thing a validator does is parse. If parsing fails, most pipelines stop there and escalate. That collapses two very different failures into one bucket:

The research question: how much of a cascade's escalation spend goes to format failures, what are the cheaper ways to fix them, and what does a failure-aware escalation policy save?

What the papers found.

"When Correct Isn't Usable" (Galeone, Park, Ettorre, and Ligorio, arXiv:2605.02363, May 2026) tested three 7-9B models plus GPT-4o on GSM8K and MATH with answers required in a JSON structure. Under a naive prompt, the small models reached up to 85% task accuracy and 0% output accuracy. A prompt that included a reference example still gave 0% output accuracy for two of the four models. The authors call it "systematic format failure": mathematically correct answers rejected for structural non-compliance. Their prompt-optimization method, AloLab, brought output accuracy to 84-87% on GSM8K and 34-40% on MATH, and 95.2% on GPT-4o. They also measured constrained decoding in their setup at 3.6x to 8.2x latency overhead, with task performance degrading in several cases.

"The Hidden Cost of Structured Generation" (Reddy, Walker, Ide, and Bedi, arXiv:2603.03305, revised June 2026) explains the degradation. Constrained decoding masks every token the grammar doesn't allow. When the model puts little probability on any valid continuation, the mask forces it down a path it wouldn't have chosen, and the reasoning suffers. Their fix, Draft-Conditioned Constrained Decoding, is training-free: generate an unconstrained draft, then run a constrained pass conditioned on that draft. On GSM8K with a 1B model it raised structured accuracy by up to 24 points, from 15.2% to 39.0%, and let smaller model pairs match or beat much larger constrained baselines.

"Repair, Not Improvement" (Lee, arXiv:2608.13959, revised September 2026) tested 0.6B to 4B models on when to call a tool and when to abstain, in English and Korean. Of 698 abstentions the constraint recovered, 545 came from outputs that had been unreadable. In the smallest Korean cell, stopping at the first line cost 20.0 points, the grammar restriction gave back 19.5, and together they netted −0.5. The author's summary is the sentence to remember: "What the restriction gives back is readable output, not judgment."

Read together, the three papers say:

1. Small models fail on format far more often than their task accuracy implies.
2. Forcing the format during generation can cost reasoning quality. Forcing it after a free draft costs much less.
3. Fixing the format recovers the answer the model already had. It never creates one it didn't have.

That third point is why format repair belongs inside the cascade, not instead of it.

The hidden cost: your router learns the wrong lesson.

The per-request cost of escalating a format failure is obvious. The larger cost is quieter.

Most teams decide which tasks can go to a cheap model by running an eval: 500 examples per task type, score the cheap model, route the task type to it if it clears a bar. If the scorer parses first and counts parse failures as wrong, the eval measures format compliance times task accuracy, not task accuracy.

Take an extraction task where the cheap model gets the right values 94% of the time, and wraps 15% of its answers in a sentence of prose or a markdown fence. Scored naively, it passes about 80%. If your bar is 90%, the whole task type goes to the flagship model. Not 20% of it. All of it.

In the cost model below, that's the difference between $9,750 and $2,941 a month for one workload. A format problem that a regex could have fixed becomes a routing decision that sends 100% of the traffic to a model five times the price. The Galeone paper is the extreme case: a scorer checking output accuracy would have rated models at 0% that were getting up to 85% of the math right.

Score content on a repaired output, and track format compliance as a separate number. One tells you whether the model can do the task. The other tells you how much repair it needs.

Methodology.

An analytical cost model, not a benchmark. The prices are September 2026 list prices. The failure split is an assumption, informed by the papers above; measure your own before trusting it.

Three policies:

• A. Flagship only. Every call to Opus 5.
• B. Cascade, escalate every failure. Haiku first; any validation failure goes to Opus 5.
• C. Cascade with triage. Haiku first; format failures get local repair, then a draft-conditioned re-shape on Haiku; content failures, persistent format failures, and repaired-but-wrong outputs go to Opus 5.

Finding 1: half of a naive cascade's bill is escalation.

Policy B looks like a good cascade. It cuts 60% against flagship-only. But look at where the money goes: $1,950 for a million Haiku calls, and another $1,950 for 200,000 Opus calls. One in five requests escalates, and each escalation costs five times what the first attempt did. Escalation is half the bill.

Policy C spends:

That's 24.6% below policy B. The escalation rate drops from 20% to 8.66%, and the retry step that makes most of that possible costs $147, about 7.5% of what it saves.

Finding 2: constrain the retry, not the first attempt.

The obvious alternative is to turn on strict structured output for every call and never see a format failure at all. For large hosted models with server-side schema enforcement, that's often the right answer, and format failures there are rare to begin with.

For small models, the papers point the other way:

• Constraining the first attempt can distort reasoning (Reddy et al.), and in the Galeone setup it added 3.6x to 8.2x latency while degrading task performance in several cases. Modern engines like XGrammar have cut per-token mask overhead sharply, so measure the latency on your own stack, but the reasoning distortion is about the mask, not its speed.
• Constraining after a free draft recovered most of the loss.

Policy C does exactly that, and only on the 7% of calls that need it. The first attempt runs unconstrained, so it reasons the way the model wants to. Only outputs that failed to parse and failed local repair get a second, short, constrained pass that copies the draft into the schema. That pass is mostly transcription, which is the easy thing for a small model.

If you do constrain the first attempt, put any reasoning field before the answer fields in your schema. Field order is generation order. A schema that asks for answer first and reasoning second makes the model commit before it thinks.

Finding 3: the triage is cheap, but it has to be strict.

The failure mode of this design is obvious, and Lee's paper names it: a repaired output is readable, and readable is not the same as right. If repair becomes a way to launder bad answers past your verifier, you've traded a cost problem for a quality problem.

So the rule is simple. Repair touches shape only. Every repaired output goes through the same content check as an output that parsed the first time. In the model, 10% of repaired outputs still fail that check and escalate, and they're included in the $2,941.

Tutorial: failure-aware escalation in Python.

The pipeline has four stages. Validate the cheap output. On a parse or schema failure, try deterministic repair. If that fails, run a draft-conditioned re-shape on the same cheap model. Send anything with a content failure, or that still won't parse, to the flagship.

Step 1: define the schema and a content check.

from pydantic import BaseModel, Field, ValidationError, field_validator

class Invoice(BaseModel):
    vendor: str
    invoice_number: str
    total: float = Field(ge=0)
    currency: str = Field(pattern=r"^[A-Z]{3}$")
    line_items: int = Field(ge=1)

    @field_validator("vendor", "invoice_number")
    @classmethod
    def not_blank(cls, v: str) -> str:
        if not v.strip():
            raise ValueError("blank")
        return v

def content_ok(inv: Invoice, source_text: str) -> bool:
    """Cheap grounding checks. Replace with your own verifier."""
    return (
        inv.invoice_number in source_text
        and f"{inv.total:.2f}".rstrip("0").rstrip(".") in source_text.replace(",", "")
    )

Keep the two checks separate. Pydantic answers "is this the right shape?" content_ok answers "is this the right answer?" Your escalation logic needs to know which one failed.

Step 2: deterministic repair.

import json, re
from json_repair import repair_json   # pip install json-repair

FENCE = re.compile(r"```(?:json)?\s*(.*?)```", re.S)

def local_repair(raw: str) -> dict | None:
    text = raw.strip()
    if m := FENCE.search(text):                  # strip markdown fences
        text = m.group(1)
    start, end = text.find("{"), text.rfind("}")
    if start != -1 and end > start:              # drop prose around the object
        text = text[start : end + 1]
    try:
        return json.loads(text)
    except json.JSONDecodeError:
        pass
    try:
        fixed = repair_json(text, return_objects=True)
        return fixed if isinstance(fixed, dict) and fixed else None
    except Exception:
        return None

This handles the boring majority: markdown fences, "Here is the JSON:" preambles, trailing commas, single quotes, unquoted keys. It costs no tokens. Be conservative. If the repair would have to invent a value, return None and let the next stage decide.

Step 3: draft-conditioned re-shape on the cheap model.

from openai import OpenAI

client = OpenAI()   # any OpenAI compatible endpoint
CHEAP, FLAGSHIP = "claude-haiku-4-5", "claude-opus-5"

RESHAPE = """You are given a draft answer that has the right information
in the wrong format. Copy it into the JSON schema below. Do not change
any value, do not add information that is not in the draft. If a field
is missing from the draft, use null.

Schema:
{schema}

Draft:
{draft}"""

def reshape(draft: str) -> dict | None:
    resp = client.chat.completions.create(
        model=CHEAP,
        messages=[{"role": "user", "content": RESHAPE.format(
            schema=json.dumps(Invoice.model_json_schema()), draft=draft)}],
        response_format={"type": "json_schema", "json_schema": {
            "name": "invoice", "schema": Invoice.model_json_schema(), "strict": True}},
        max_tokens=300,
    )
    return local_repair(resp.choices[0].message.content or "")

This is the draft-conditioned idea from Reddy et al. in its simplest API form. The first attempt reasoned freely; this pass only transcribes. It's short, it runs on the cheap model, and it's the only call in the pipeline that uses constrained output. If your provider doesn't support json_schema for the cheap model, drop response_format and rely on the repair function; the prompt does most of the work.

Step 4: the escalation decision.

from dataclasses import dataclass

def call(model: str, prompt: str, doc: str, strict: bool = False) -> str:
    kwargs = {}
    if strict:
        kwargs["response_format"] = {"type": "json_schema", "json_schema": {
            "name": "invoice", "schema": Invoice.model_json_schema(), "strict": True}}
    resp = client.chat.completions.create(
        model=model, max_tokens=400,
        messages=[{"role": "system", "content": prompt},
                  {"role": "user", "content": doc}], **kwargs)
    return resp.choices[0].message.content or ""

def try_parse(raw: str) -> dict | None:
    try:
        data = json.loads(raw)
        return data if isinstance(data, dict) else None
    except json.JSONDecodeError:
        return None

@dataclass
class Result:
    invoice: Invoice
    path: str   # "first_pass" | "repaired" | "reshaped" | "escalated"

def extract(doc: str, prompt: str) -> Result:
    raw = call(CHEAP, prompt, doc)

    for stage, attempt in (("first_pass", lambda: try_parse(raw)),
                           ("repaired",   lambda: local_repair(raw)),
                           ("reshaped",   lambda: reshape(raw))):
        data = attempt()
        if data is None:
            continue                          # format failure, try next stage
        try:
            inv = Invoice.model_validate(data)
        except ValidationError:
            continue                          # schema failure, try next stage
        if content_ok(inv, doc):
            return Result(inv, stage)
        break                                 # readable but wrong: escalate now

    inv = Invoice.model_validate_json(call(FLAGSHIP, prompt, doc, strict=True))
    return Result(inv, "escalated")

The break is the important line. A content failure never gets a repair attempt, because repair can't fix it. It goes straight to the flagship, the same as it would in policy B. Only shape problems get the cheap path.

Step 5: log the path.

Count path per task type every day. It gives you three numbers a plain pass/fail rate hides:

• First-pass rate: how often the cheap model is right and well-formed.
• Repair rate: how often it's right but badly formed. If this is high, fix the prompt. The Galeone results suggest a better prompt alone can take output accuracy from 0% to the mid-80s.
• Escalation rate: what you actually pay the flagship for. This is the number to drive down, and the one to use when you decide whether a task type belongs on the cheap tier.

When not to bother.

• Your cheap tier already has server-side strict mode, and format failures are under 1%. Then the triage saves little. Keep the separate counters anyway, so you'd notice if it changed.
• Free-form output. Summaries and drafts have no format to fail. This is a structured-output problem.
• Tool calls in an agent loop. A malformed tool call is a format failure, and the same triage works. But a tool call that abstains when it shouldn't, or fires when it shouldn't, is a judgment failure, and Lee's paper is specifically about how a grammar can hide that. Verify the decision, not just the arguments.
• Low volume. At 10,000 calls a month, policy B's extra spend over C is about $10. Ship the simple cascade and revisit when the bill is worth it.

Where Nadir fits.

The triage in this post is code in your own pipeline. It doesn't require a gateway. What it depends on is the decision before it: which requests go to the cheap tier at all. That's what Nadir makes. POST /v1/bucket classifies a prompt's difficulty without spending provider tokens, and Nadir's verifier-gated cascade checks the cheap answer before deciding to escalate, rather than retrying blind. Through the OpenAI compatible gateway, every request reports what it cost and which model served it, so you can see your own escalation rate instead of using our assumed 20%.

If you've concluded from an eval that a task "needs" the flagship model, it's worth re-running that eval with the answers repaired before scoring. Start with a free key, bucket a day of the traffic you route to the expensive tier, and look at how much of it was only ever a formatting problem.

Conclusion.

A cheap model that gets the answer right in the wrong format isn't a failed cheap model. It's a formatting bug. Treating it as a failure costs you twice: once per request, when a flagship call fixes a comma, and once per task type, when an eval that can't tell shape from substance sends everything to the expensive tier. The 2026 research gives a clear order of operations. Let the small model reason freely. Repair shape locally. Re-shape on the same cheap model, conditioned on its own draft. Check content every time, and escalate only when the content is wrong. On this post's model, that cuts escalations from 20% of traffic to under 9%, and none of the saving comes from accepting a worse answer.

Costs in this post are illustrative, modeled for this post from public list prices, and are not derived from customer data. Sources: [Galeone, Park, Ettorre, and Ligorio, "When Correct Isn't Usable: Improving Structured Output Reliability in Small Language Models," arXiv:2605.02363, May 2026](https://arxiv.org/abs/2605.02363). [Reddy, Walker, Ide, and Bedi, "The Hidden Cost of Structured Generation in LLMs: Draft-Conditioned Constrained Decoding," arXiv:2603.03305, 2026](https://arxiv.org/abs/2603.03305). [Lee, "Repair, Not Improvement: Decomposing Constrained Decoding in Tool-Call Abstention," arXiv:2608.13959, 2026](https://arxiv.org/abs/2608.13959). [json-repair on PyPI](https://pypi.org/project/json-repair/). [Anthropic, Pricing](https://platform.claude.com/docs/en/about-claude/pricing).