The JSON Tax

The overhead you added without noticing.

You added structured outputs because they solved a real problem. Your LLM calls used to return strings you had to parse, and sometimes the JSON was malformed, and you had retry logic, and some days the whole thing broke in production. Structured outputs fixed that. The JSON comes back clean, the model validates, and the downstream code never hits a parse error.

What you did not notice is that your token bill quietly increased. Not dramatically, but consistently, on every single call where you use it. The overhead has two sources, they compound at the output token rate, and most teams have never measured either one.

The first tax: schema tokens you send on every call

When you use structured outputs, whether that is OpenAI's response_format with a JSON schema, Anthropic's tool calling for structured responses, or Google Gemini's response_schema, you send a schema definition with every API call. The model needs this schema to generate a valid, schema-conforming response. That schema is not free.

A minimal schema defining a single object with five string fields runs 150 to 300 tokens. A realistic production schema, an entity extraction pipeline pulling names, dates, addresses, and line items from documents, runs 600 to 1,500 tokens. A complex schema with nested objects, enum constraints, and field descriptions runs 2,000 to 4,000 tokens.

Here is what a schema for a moderate document extraction task looks like:

{
  "type": "object",
  "properties": {
    "vendor_name": {
      "type": "string",
      "description": "The legal name of the vendor as it appears on the document"
    },
    "invoice_number": {
      "type": "string",
      "description": "The invoice or document reference number"
    },
    "invoice_date": {
      "type": "string",
      "description": "The invoice date in ISO 8601 format"
    },
    "line_items": {
      "type": "array",
      "items": {
        "type": "object",
        "properties": {
          "description": { "type": "string" },
          "quantity":    { "type": "number" },
          "unit_price":  { "type": "number" },
          "total":       { "type": "number" }
        },
        "required": ["description", "quantity", "unit_price", "total"]
      }
    },
    "subtotal":      { "type": "number" },
    "tax_amount":    { "type": "number" },
    "total_amount":  { "type": "number" },
    "payment_terms": { "type": "string" }
  },
  "required": ["vendor_name", "invoice_number", "invoice_date",
               "line_items", "total_amount"]
}

That schema runs approximately 380 tokens. Every call. Whether the invoice has one line item or thirty. Whether the request succeeds or fails. Whether the model uses every field or only three.

At 100,000 daily API calls on Claude Opus 4.8 at $5 per million input tokens:

• 380 schema tokens times 100,000 calls = 38 million input tokens per day
• At $5/M: $190 per day
• Annualized: $69,350 per year

In schema tokens alone. Not the document content, not the system prompt, not the user query. Just the schema definition you wrote once and never think about again.

The second tax: JSON is verbose at output rates

The less obvious cost is on the output side.

When a model generates a structured JSON response, it produces more tokens than it would for equivalent prose. The JSON framing, keys, colons, quotes, braces, brackets, and commas, takes tokens that plain text does not. That matters because output tokens cost five times more than input tokens on most frontier models. At Opus 4.8: $5/M input, $25/M output. At GPT-5.5: $12.50/M input, $30/M output.

Consider two equivalent responses to the same invoice extraction request:

Plain text (about 63 tokens):

Acme Corp, INV-4821, Feb 14 2026. Line items: consulting $4,500,
software license $1,200, support retainer $800. Subtotal $6,500,
tax $650, total $7,150. Net 30.

Structured JSON (about 128 tokens):

{
  "vendor_name": "Acme Corp",
  "invoice_number": "INV-4821",
  "invoice_date": "2026-02-14",
  "line_items": [
    {"description": "Consulting services", "quantity": 1,
     "unit_price": 4500.00, "total": 4500.00},
    {"description": "Software license",    "quantity": 1,
     "unit_price": 1200.00, "total": 1200.00},
    {"description": "Support retainer",    "quantity": 1,
     "unit_price": 800.00,  "total": 800.00}
  ],
  "subtotal": 6500.00,
  "tax_amount": 650.00,
  "total_amount": 7150.00,
  "payment_terms": "Net 30"
}

Same information. Twice the output tokens. Priced at the 5x output rate. In practice the overhead varies by schema complexity: 20 to 30% for simple extractions, 40 to 60% for deeply nested structures or arrays with many items.

What this costs at scale

Take a document processing pipeline at 100,000 daily calls. Average output without structured output: 800 tokens. With a moderately complex JSON schema, output averages about 1,000 tokens, a 25% increase.

At 500,000 daily calls, common for enterprise document processing or agentic pipelines, the total crosses $1.25 million per year. At one million calls, it exceeds $2.5 million. None of this appears as a line item in your billing dashboard. Schema tokens look like input tokens. JSON verbosity looks like output tokens. No dashboard breaks them out unless you specifically instrument for it.

When the overhead is worth paying

Structured outputs solve real problems. Reliable JSON parsing eliminates retry logic, simplifies downstream code, and removes an entire category of production failures. The overhead is often a fair trade.

The issue is not using structured outputs. The issue is using them when you do not need them, without knowing what it costs. Here are the cases where the overhead is not justified:

• Single-value responses. If your pipeline asks whether a document is an invoice or a purchase order, a plain string response with clear instructions ("respond with exactly invoice or purchase_order") is cheaper, faster, and equally reliable for binary classification.
• Internal pipeline steps that are never machine-consumed. If you are extracting a field to display in a UI, brief text extraction with prompt-based guidance is sufficient. Schema validation only matters when the output feeds a parser.
• High-volume simple queries. A customer service router classifying intents into one of ten categories runs 10,000 to 100,000 times per day. Even a 200-token schema multiplied across that volume costs $365 per year per schema, for a task that a regex or a simple text instruction handles equally well.
• Calls that already have high output token counts. A 3,000-token generation task with 25% JSON overhead adds 750 tokens at $25/M. At scale, that is the most expensive line item in your pipeline, and it is invisible.

How to measure your own JSON tax

The measurement takes about an hour:

import anthropic

client = anthropic.Anthropic()

# With structured output
r_structured = client.messages.create(
    model="claude-opus-4-8",
    max_tokens=1024,
    tools=[{
        "name": "extract_invoice",
        "description": "Extract invoice fields",
        "input_schema": YOUR_SCHEMA
    }],
    tool_choice={"type": "tool", "name": "extract_invoice"},
    messages=[{"role": "user", "content": YOUR_DOCUMENT}]
)

# Without structured output
r_text = client.messages.create(
    model="claude-opus-4-8",
    max_tokens=1024,
    system="Extract vendor_name, invoice_number, invoice_date, and "
           "total_amount from the invoice. Return only valid JSON.",
    messages=[{"role": "user", "content": YOUR_DOCUMENT}]
)

print(f"Structured: {r_structured.usage.input_tokens} in, "
      f"{r_structured.usage.output_tokens} out")
print(f"Text:       {r_text.usage.input_tokens} in, "
      f"{r_text.usage.output_tokens} out")

Run this against 50 real documents from your production dataset. The difference in usage numbers is your JSON tax. Multiply by your daily call volume and your model's pricing to get the annualized overhead. Teams that run this audit for the first time typically find that structured output overhead accounts for 8 to 22% of their total monthly input plus output bill on extraction pipelines.

Reducing the overhead without giving up reliability

Compress your schemas. Remove field descriptions where the property name is self-explanatory. Replace verbose names with short equivalents when you control both sides. Remove required arrays when you validate in application code rather than relying on schema enforcement. A 1,200-token schema often compresses to 400 to 600 tokens with no loss in parsing reliability.

Before:

{
  "type": "object",
  "properties": {
    "vendor_name": {
      "type": "string",
      "description": "The legal name of the vendor or supplier company
                      as it appears on the document header"
    },
    "invoice_number": {
      "type": "string",
      "description": "The unique invoice identifier or reference number"
    }
  },
  "required": ["vendor_name", "invoice_number"]
}

After:

{"type":"object","properties":{"vendor":{"type":"string"},
"inv_num":{"type":"string"}}}

Same extraction quality. Roughly 70% fewer schema tokens.

Apply structured output selectively. Not every call in a pipeline needs schema enforcement. Run a classifier before applying schemas: calls that produce machine-consumed output get the schema; calls that produce informational or free-form content skip it. This is exactly the pattern Nadir implements, the router identifies which calls in your pipeline genuinely require schema enforcement and only applies the overhead where it is justified.

Route schema-heavy calls to cheaper models. Structured output support is now standard across providers. The $1/M input models handle schema-conforming extraction as reliably as Opus on most document types. For high-volume extraction pipelines that run the same schema on every call, routing alone typically saves more than schema compression. The same overhead shows up in the tool schemas function calling sends on every agentic call.

Cache your schemas at the API level. For repeated use of the same schema across calls, prompt caching stores the schema tokens at roughly a 90% discount on subsequent reads. A 1,000-token schema cached across 100,000 daily calls saves roughly $450 per day versus re-sending it uncached. The implementation is a single cache control header.

Use `max_tokens` to cap JSON verbosity. If your schema has a bounded output size, set max_tokens to a tight ceiling. A schema that can produce at most 400 tokens of JSON should not have max_tokens=4096. The model will not generate extra tokens if it reaches the natural end of the JSON object, but an explicit ceiling prevents runaway output on edge cases and keeps your worst-case bill predictable.

The practical audit checklist

If you are using structured outputs in production, run this audit:

1. Pull one day of API logs with input and output token counts per call.
2. Flag every call that includes a response_format, tool_choice, or equivalent schema enforcement parameter.
3. Average the token counts for schema calls versus non-schema calls of similar task complexity.
4. Calculate the markup percentage on both input and output separately.
5. Multiply by your daily call volume and annual rate to get the annual overhead.
6. Identify which schema calls are not actually consumed by a downstream parser, those are the first candidates to eliminate.

In teams that have run this audit, structured output overhead accounts for 8 to 22% of total monthly API spend for mixed workloads, and 25 to 35% for extraction-heavy pipelines where nearly every call uses a schema. For the calls that legitimately need reliable JSON, the overhead is a reasonable price. For the calls that are using structured outputs as a habit rather than a necessity, it is pure waste.

The pattern is the same one that shows up across every LLM cost category: the overhead is real, it compounds at scale, and it is completely invisible in standard billing dashboards. Schema tokens look like input tokens. JSON verbosity looks like output tokens. No dashboard breaks them out. Measurement reveals them. Routing and compression address them. Most teams have never done either.

Sources: [Anthropic, Claude Pricing, 2026](https://www.anthropic.com/pricing). [OpenAI, API Pricing, 2026](https://openai.com/api/pricing). [Google, Gemini API structured output docs](https://ai.google.dev/gemini-api/docs/structured-output). [Anthropic, "Tool use," Anthropic Docs](https://docs.anthropic.com/en/docs/build-with-claude/tool-use). [Anthropic, "Prompt Caching," Anthropic Docs](https://docs.anthropic.com/en/docs/build-with-claude/prompt-caching). Token counts are illustrative estimates from the schemas shown; measure against your own workload.