Forget the Thinking

TL;DR.

A reasoning agent writes its thinking once and then pays to carry it on every later turn. A paper posted to arXiv on September 24, "When Can Agents Forget Their Reasoning?", tested what happens if you stop carrying it. The authors kept every action, tool call and tool output intact and deleted only the old reasoning.

• Deleting all past thinking cut tokens 62.9% and raised reward from 62.67 to 71.78 on an 80-task subset of WorkBuddyBench. Every pruning variant they tried scored above the full-history agent.
• Their method, also called ICLR (no relation to the conference), removes the 80% of each step's reasoning blocks a small proxy model finds most predictable. On all 260 tasks it lifted average reward from 69.9 to 71.8 and cut total tokens 18.5%.
• The average hides a split. Security tasks gained 22.9 points. Code tasks lost 6.4 points and used more tokens, because the agent re-derived what it had forgotten.
• When you forget matters as much as what. In our simulation of a 40-step agent on Sonnet 5.5 prices, dropping each step's reasoning as it's appended cut the bill 20.1%. Keeping only the latest step's reasoning cut tokens 37% and raised the bill 0.9%, because it rewrote cached history every turn.
• Below is forget.py, 47 lines that drop a step's reasoning at append time, and only once the step has written its result somewhere the agent can reread.

The paper's results come from one acting model, DeepSeek V4 Flash, on one benchmark. The cost simulation is ours, at October 2026 list prices, and is an illustration, not a measurement. No figure here comes from Nadir customer data.

What the paper did.

Long-horizon agents accumulate reasoning. Each step, the model thinks, calls a tool, reads the result, and the whole exchange goes into history. By step 40, most of the context is old thinking about decisions that were already carried out.

Mingxuan Wang and colleagues asked when that history can go. Their method runs after each completed step:

1. Split the step's reasoning into blocks at blank lines.
2. Score each block with a frozen Qwen3.5 9B model: mean token entropy, conditioned on the system prompt, tools and history so far. Low entropy means the block is predictable from what's already in context.
3. Replace the lowest-entropy 80% of blocks with [SKIP]. Keep the rest in order.
4. Never touch actions, tool arguments, observations or answers.

It's training-free, and the acting agent never sees the scorer. The benchmark, WorkBuddyBench, has 260 long-horizon tasks across Code (80), Office (50), Security (60) and Web (70).

The ablation is the most useful table in the paper, because it compares the clever method against blunt ones.

Two things stand out. Every variant, including random deletion, beat the full-history agent, which says old reasoning was actively hurting this model, not just costing money. That matches the long-context degradation we covered in context rot. And the savings don't scale with how much you delete: random deletion at 10% saved more tokens than at 20%. The paper calls this trajectory amplification. A small edit changes the next tool call, which changes the next observation, which changes when the agent decides it's done.

The split under the average.

On the full 260 tasks, ICLR won 102 tasks, tied 76 and lost 82 against the base agent. The per-domain numbers explain why.

Code is the case to understand, because it's where most teams run agents. Forgetting reasoning in Code made the agent worse and more expensive. It thought its way back to conclusions it had already reached. In a controlled Office run, deleting all reasoning raised reasoning tokens from 1,817 to 7,491 while the score fell from 0.87 to 0.81. A four-fold re-derivation is far past any saving from a shorter context.

The paper's explanation is its best finding: reasoning is safe to forget once its result lives somewhere else. After the agent wrote code to disk, keeping or clearing the thinking that produced it gave the same score, 0.7587 either way. When a tool error stayed visible in the context, the agent fixed the problem directly in 9 of 9 runs. When it was hidden, the agent re-ran the command to see the error again in 9 of 9. Across their probes, the risk of a later replan or re-derivation was 67.3% when the derived state existed only in reasoning and 36.2% when it had been written to a file, tool output or the environment.

That's a rule you can implement without an entropy model: drop a step's reasoning after the step has externalized what it concluded.

Why when you forget changes the bill.

A token you delete isn't always a dollar you save. With prompt caching, every turn re-sends the history, but most of it is billed as a cache read. Rewriting earlier history breaks the cached prefix and bills everything after the edit as a cache write. We covered the mechanics in the compaction tax and, with a proxy that names every miss, in Watch the Wire.

ICLR prunes each step once, after it completes and before it enters history, so history stays append-only. Its 33.3% drop in cache-read tokens is what an append-only edit looks like: less to read, nothing to rewrite.

To see what the timing does to a bill, we simulated a 40-step agent at Sonnet 5.5 list prices ($2 input, $2.50 cache write, $0.10 cache read, $10 output per million tokens). Each step produces 1,500 reasoning tokens and a 150-token action, and gets back a 1,200-token observation, on top of a 20,000-token tools and system prompt. Four policies:

"Keep only the latest" sounds like the safe middle ground. The paper even found it scored the same as keeping everything in a controlled test. But it edits the previous step on every turn, so every turn pays a fresh cache write for the tail of the history. It sends 37% fewer tokens and costs slightly more. Same idea, wrong moment.

Two more numbers from the simulation set the stakes:

• Carrying reasoning cost $0.26 on a $1.28 run. The reasoning itself, billed once as output, was $0.60. Carrying it added a cache write when it entered history and a cache read on every later turn. Dropping it saves that 20%, not the output.
• Dropping breaks even at 43% more reasoning. If forgetting makes the model think 43% longer per step, the savings are gone. The paper's Code domain (+16% output) stays under that line. Its controlled Office case (4x reasoning) is far over it. Measure output tokens per step before and after, per task type, or you'll miss the case where it backfires.

The same simulation at Claude Fable 5.1 prices (0 input, 2.50 cache write, /bin/bash.25 cache read, 0 output) moves every line the wrong way. Dropping at append saves 17.8% instead of 20.1%, breaks even at 34% more reasoning instead of 43%, and keeping only the latest step costs 6.2% more instead of 0.9%. Output and cache writes are 5x Sonnet's rates there, while cache reads are only 2.5x. The more expensive the model, the less room forgetting has to backfire.

Tutorial: forget.py.

This implements the rule the paper's evidence supports, not its entropy scorer: when a step finishes, drop its reasoning if the step wrote something durable (a successful file write or edit) or produced an error the agent can see. Otherwise keep it. Edits happen once, at append time, so the cached prefix is never rewritten.

It targets OpenAI compatible APIs that return reasoning as a separate reasoning_content field, as DeepSeek's does. Check your provider's rules before using it elsewhere: some APIs require reasoning blocks to be passed back unchanged within a tool-use loop, and Anthropic signs its thinking blocks, so they can be omitted only where the docs allow and never edited.

"""forget.py: drop an agent step's reasoning once, when the step is appended.

Works with OpenAI compatible chat APIs that return reasoning in a separate
`reasoning_content` field on the assistant message. Actions, tool arguments
and tool outputs are never edited, only the reasoning text.
"""
import re

WRITES = re.compile(r"write|edit|patch|create|save|apply|commit", re.I)


def externalized(assistant, tool_results):
    """True when this step left its conclusion somewhere the agent can reread:
    a successful file write or edit, or a tool error the agent can see."""
    calls = assistant.get("tool_calls") or []
    wrote = any(WRITES.search(c["function"]["name"]) for c in calls)
    ok = all(not r.get("is_error") for r in tool_results)
    errored = any(r.get("is_error") for r in tool_results)
    return (wrote and ok) or errored


def forget(assistant, tool_results, policy="after_artifact"):
    """Return the assistant message to store in history for this step."""
    msg = dict(assistant)
    reasoning = msg.pop("reasoning_content", None)
    if not reasoning:
        return msg
    if policy == "keep":
        msg["reasoning_content"] = reasoning
    elif policy == "drop":
        pass  # nothing kept; the action and tool calls say what was decided
    elif policy == "after_artifact":
        if not externalized(assistant, tool_results):
            msg["reasoning_content"] = reasoning
    else:
        raise ValueError(policy)
    return msg


def append_step(history, assistant, tool_results, policy="after_artifact"):
    """The only place history grows. Earlier messages are never rewritten,
    so the cached prefix from the last request stays valid."""
    history.append(forget(assistant, tool_results, policy))
    for r in tool_results:
        history.append({"role": "tool", "tool_call_id": r["tool_call_id"],
                        "content": r["content"]})
    return history

Wire it into the loop where you'd otherwise history.append(...):

from openai import OpenAI
from forget import append_step

client = OpenAI(base_url="https://api.deepseek.com", api_key=KEY)
history = [{"role": "system", "content": SYSTEM}, {"role": "user", "content": task}]

while True:
    resp = client.chat.completions.create(model=MODEL, messages=history, tools=TOOLS)
    msg = resp.choices[0].message.model_dump(exclude_none=True)
    if not msg.get("tool_calls"):
        break
    results = [run_tool(c) for c in msg["tool_calls"]]  # each: tool_call_id, content, is_error
    append_step(history, msg, results, policy="after_artifact")

Run the same task set three times, with policy="keep", "drop" and "after_artifact", and log three numbers per task: pass or fail, output tokens per step, and billed cost. The second number is the one that catches re-derivation. If drop passes as often as keep with flat output per step, use it. If output per step climbs on some task types, use after_artifact for those, or keep reasoning there entirely.

Three things the script deliberately doesn't do:

• It doesn't edit earlier steps. Retroactive pruning is where the "keep latest" policy lost money. If you need to shrink older history, do it once at a natural break and accept one cache write, as in the cache write tax.
• It doesn't score blocks. The entropy scorer bought ICLR about two reward points over deleting everything, at the cost of running a 9B model on every step. For most teams the externalization rule is the cheaper first experiment.
• It doesn't touch the current step. Within a tool-use loop, the step in progress keeps its reasoning. Only completed steps are pruned.

How this fits with other context compression.

The EMNLP 2026 survey "Context Compression for LLM Agents" by Yifei Wang and colleagues sorts failures into three kinds: deciding wrongly what to compress, losing information while compressing, and failing to get it back afterward. Forgetting reasoning has a useful property on all three. The decision rule is simple, the actions and observations that carry the facts are never lossy, and the agent can recover state by rereading a file or re-running a tool.

The paper ran several of these as baselines on its 40-task pilot set. A recency window of five steps scored 56.4 and used 56% more tokens than full history. LLMLingua-2 scored 62.0 and used 27% more. Both show the same amplification: cut the wrong context and the agent spends more tokens getting it back. We looked at this backfire for prompt compression in when compression expands output.

Where Nadir fits.

Nadir works on the request, not inside your agent loop, so it doesn't decide which reasoning your agent keeps. forget.py belongs in your harness. Two parts of Nadir sit next to it.

Routing decides how expensive each step's thinking is. With model="auto", Nadir sends each request to the cheapest model in your configured set that clears your quality floor. A simple step routed to a smaller model produces cheaper reasoning, and cheaper reasoning costs less to carry. The simulation above shows why that matters: on Fable 5.1, a step's reasoning costs five times as much to produce, write and re-derive as on Sonnet 5.5.

Context Optimize trims the rest of the request deterministically. In safe mode it deduplicates repeated tool schemas and system prompts, minifies embedded JSON and normalizes whitespace, with no model calls. Because it's a deterministic function of the input, the same prefix compresses the same way every turn, so it doesn't break the cache the way a query-aware compressor can. It's off by default and trims conversations past 40 user turns. The details are in the Context Optimize writeup.

Every response reports the model that served it and what it cost, so the per-task comparison in the tutorial doesn't need a separate logging setup. If you'd rather measure before anything changes, Observe mode serves exactly the model you asked for and records what routing would have chosen. Savings depend on your workload, and nothing in this post forecasts yours.

Start with a free key, and run forget.py's three policies against your own tasks first.

Conclusion.

An agent's reasoning is working memory, and the paper's evidence says to treat it that way. Once a step has written its conclusion into a file, a tool result or a visible error, the thinking that produced it is mostly dead weight: on one model and one benchmark, deleting all of it cut tokens 63% and scored higher. Where the conclusion lived only in the thinking, as it often did in Code tasks, forgetting made the agent work it out again, at a higher bill. Prune at append, prune after the artifact exists, never rewrite history you've already cached, and measure output per step to catch the case where forgetting costs more than remembering.

Paper results are from Wang et al. (arXiv:2609.29875, v1, September 24, 2026), Tables 1, 2, 3 and 7 and the controlled interventions, on one acting model (DeepSeek V4 Flash) and one benchmark. The 80-task, 40-task and 260-task results are different subsets and aren't directly comparable. The cost simulation is ours: a 40-step agent with an append-only cached prefix, priced at October 2026 list rates for Claude Sonnet 5.5 and Claude Fable 5.1, from our earlier posts on [Fable 5.1 vs Astra](/blog/same-sticker-different-bill-cache-read-pricing-fable-astra) and [Watch the Wire](/blog/run-your-own-llm-proxy-cache-miss-claude-code-cost). No figure is derived from customer data. Sources: [Mingxuan Wang et al., "When Can Agents Forget Their Reasoning? ICLR for Long-Horizon Agent Context Compression", arXiv:2609.29875](https://arxiv.org/abs/2609.29875). [Yifei Wang et al., "Context Compression for LLM Agents: A Survey of Methods, Failure Modes, and Evaluation", EMNLP 2026](https://aclanthology.org/2026.emnlp-main.2396/). [Microsoft Research, ACON: Optimizing Context Compression for Long-horizon LLM Agents](https://www.microsoft.com/en-us/research/publication/acon-optimizing-context-compression-for-long-horizon-llm-agents/). [Anthropic, Prompt caching](https://docs.anthropic.com/en/docs/build-with-claude/prompt-caching).