Karpathy renamed the job in one tweet
In June 2025, Andrej Karpathy posted that he preferred "context engineering" over "prompt engineering." His reasoning: "people associate prompts with short task descriptions you'd give an LLM in your day-to-day use. When in every industrial-strength LLM app, context engineering is the delicate art and science of filling the context window with just the right information for the next step." Source: Andrej Karpathy, X post, June 2025.
That is a narrow-sounding complaint about word choice. It is not one. "Prompt" implies the bottleneck is phrasing, the sentence you type before the model answers. In a real production system, the model never sees your prompt in isolation. It sees a context window assembled at runtime from a system prompt, retrieved documents, tool schemas, conversation history, memory, and the output of whatever the agent did three steps ago. Calling the discipline "prompt engineering" points everyone at the wrong lever. The tweet went to the top of Hacker News within a day. Source: Hacker News discussion, June 2025.
Three months later, Anthropic made the renaming official with an engineering guide of its own: Effective context engineering for AI agents, published September 2025. Its framing: "context must be treated as a finite resource with diminishing marginal returns," the same way human working memory is finite. The guide introduces an idea worth sitting with: an attention budget. Every token you add to the window is not free just because the window is large enough to hold it. It draws down a budget the model spends on deciding what to weight.
Why now, not two years ago
Nothing about attention mechanics changed in 2025. Three things happened at once that made the old habit, when unsure whether something is relevant, include it anyway, expensive in a way it had not been before.
Context windows got large enough to make that habit feel free. Gemini 2.5 Pro shipped a 1-million-token window. The implicit marketing promise of a bigger window is that it is strictly better than a smaller one. It is not. This blog has already covered the research showing why: Chroma tested 18 frontier models from 1,000 to 1,000,000 tokens, and every one of them got measurably worse before hitting its context limit. Context rot is what happens when "the window can hold it" gets treated as the same question as "the model should read it."
Agent loops multiplied the number of times that habit gets paid for. A single agent step already reruns the entire conversation history through the model on every turn. Multi-agent orchestration multiplies that same overhead roughly 15 times, and a research agent can spend 82% of its token budget on structural overhead before writing a word of the actual answer. None of that is one bloated prompt. It is the same undisciplined context assembly, repeated at every hop.
And the failure mode got a name and a mechanism, not just a vibe. Chroma's methodology extended the classic needle-in-a-haystack test with distractors and low-similarity questions, and found the decline was steepest exactly where production agents are weakest: long histories full of plausible-but-irrelevant tool output. Source: Chroma Research, "Context Rot," 2025. The quality problem and the cost problem turned out to be the same tokens.
The taxonomy that stuck: write, select, compress, isolate
Around the same time, LangChain's Lance Martin published a working taxonomy for the discipline: context engineering is "the delicate art and science of filling the context window with just the right information at each step of an agent's trajectory," broken into four strategies: write, select, compress, and isolate. Source: Lance Martin, "Context Engineering for Agents," June 23, 2025. It is a useful taxonomy precisely because it is not a new idea. Every strategy in it already has a cost number attached, mostly from research this blog has already covered.
| Strategy | What it means | Already measured here |
|---|---|---|
| Write | Persist state outside the window instead of re-sending it in full every turn | Anthropic's compaction, see below |
| Select | Retrieve only what a step needs, just in time, instead of loading everything up front | RAG pipelines fetch 20 chunks, the model reads 3 |
| Compress | Summarize, prune, or losslessly re-encode whatever must stay in the window | Prompt compression cuts input bills 3 to 5x in production |
| Isolate | Give each sub-agent its own clean, scoped context instead of one shared, sprawling one | Sub-agent results condense to 1,000 to 2,000 tokens, see below |
Three of those four rows are old news on this blog under different names. What is new, and worth its own section, is what Anthropic's guide adds to "write" and "isolate": a specific mechanism for each, not just a principle.
Compaction and progressive disclosure: the part with an actual recipe
Compaction. When a session nears its context limit, summarize the conversation and reinitialize with the condensed version instead of the full transcript. Anthropic's guide is specific about what a good compaction step keeps and drops: preserve architectural decisions, unresolved bugs, and implementation details still in play, and discard redundant tool outputs and superseded steps. Source: Anthropic, "Effective context engineering for AI agents," September 2025. This is the same principle behind multi-turn conversations billing your first message 20 times over, except compaction is the fix applied proactively instead of discovered on an invoice.
Structured note-taking. An agent maintains a persistent scratchpad, a to-do list, or a memory file outside the context window and reads back only the piece it needs for the current step. This is how a long-running coding agent tracks a 50-step plan without carrying all 50 steps in every turn's input. The note-taking is cheap. Re-sending the entire plan every turn is not.
Just-in-time context. Instead of loading a file's full contents, load a reference, a file path, an identifiers, and fetch the content only when a step actually needs it. Anthropic calls this progressive disclosure: the agent discovers relevant context incrementally rather than front-loading everything it might conceivably need. It is the same shape of fix as a browser agent switching from a full accessibility-tree dump to compact element references, a 55x spread in tokens for reading the identical page.
Sub-agent isolation. A specialized sub-agent gets its own clean context window, does the exploration or research, and hands back a condensed result, typically 1,000 to 2,000 tokens, to the coordinating agent. Source: Anthropic, "Effective context engineering for AI agents," September 2025. The coordinator's context stays small no matter how much scratch work the sub-agent did to get there. That is the mechanism, not just the principle, behind the isolation gains this blog measured in multi-agent orchestration.
A context budget, sketched
None of this requires a framework. It requires a function that decides, on every turn, what earns a place in the window before the window fills up on its own:
# Allocate a fixed token budget across context sections, in priority order
def assemble_context(system_prompt, tools, memory, retrieved_docs, history, budget):
context = []
remaining = budget
# Fixed costs first: the model can't function without these
context.append(system_prompt)
remaining -= count_tokens(system_prompt)
context.append(select_tools(tools, task)) # select: only the tools this step needs
remaining -= count_tokens(context[-1])
# Write: pull in only the memory relevant to the current step, not the whole log
relevant_memory = select_relevant(memory, task, top_k=3)
context.append(relevant_memory)
remaining -= count_tokens(relevant_memory)
# Select + compress: retrieve narrowly, then shrink what's kept
top_docs = retrieve(retrieved_docs, task, top_k=3)
compressed_docs = compress(top_docs, ratio=0.3)
context.append(compressed_docs)
remaining -= count_tokens(compressed_docs)
# Write: compact history once it stops fitting cleanly in what's left
if count_tokens(history) > remaining:
history = compact(history, keep="decisions_and_open_threads")
context.append(history)
return contextThat is the whole idea. Every one of the four strategies shows up as a decision inside one function: what gets selected, what gets compressed, what gets written to memory instead of resent, and what gets isolated into a sub-agent call that returns a summary instead of a transcript.
What to ship this week
- Name your context budget before you write the rest of the code. Pick a token ceiling per turn and treat every section, system prompt, tools, memory, retrieval, history, as competing for a slice of it, not as free just because the window is large enough.
- Retrieve narrowly, not broadly. If your RAG pipeline fetches more chunks than the model actually uses, that gap is pure waste and pure distractor risk. Measure it directly.
- Compact proactively, not reactively. Summarize and reinitialize before a session hits its limit, not after the model starts silently getting worse.
- Isolate exploration from coordination. A sub-agent's scratch work should never reach the coordinator in full. Anthropic's own systems return 1,000 to 2,000 tokens regardless of how much a sub-agent explored to produce them.
- Treat context rot as a cost signal, not just a quality one. The tokens degrading your accuracy and the tokens inflating your bill are the same tokens. Fixing one fixes both.
Where routing fits in a discipline about context
Context engineering answers what goes into the call. It does not answer which model the call goes to, and that is a second, independent axis on the same bill. A well-engineered context sent to a frontier model on every request is still spending more than it needs to on the model side. Nadir routes each request to the cheapest model that can handle it once the context is already built, and Context Optimize applies the compress strategy from this taxonomy automatically, no LLMLingua integration to wire up yourself. The two axes multiply the same way compression and routing already do: engineer what you send, then route it to the model that should read it.
Conclusion
Karpathy's complaint was about a word, but the word was hiding the actual engineering problem. "Prompt" suggests the fix is phrasing. "Context" names the real object: a runtime-assembled window with a finite attention budget, filled from a handful of sources that all compete for space. Anthropic gave that object a mechanism, compaction, structured notes, progressive disclosure, sub-agent isolation. LangChain gave it a taxonomy that makes the sources easy to audit one at a time. None of the four strategies is new to a reader of this blog. What is new is having one name for all of them, and a budget to hold them to.
Related reading
- Chroma tested 18 frontier models from 1,000 to 1,000,000 tokens. Every one of them got measurably worse before hitting its context limit.
- Your RAG pipeline fetches 20 chunks per query. The model reads 3.
- Multi-agent AI costs 15x more, and almost nobody routes it.
- Your system prompt grew from 500 to 8,000 tokens and nobody noticed.
- Compression saves you once. Routing plus compression saves you twice.
Sources: [Andrej Karpathy, X post, June 2025](https://x.com/karpathy/status/1937902205765607626). [Hacker News discussion](https://news.ycombinator.com/item?id=44379538). [Anthropic, "Effective context engineering for AI agents," September 2025](https://www.anthropic.com/engineering/effective-context-engineering-for-ai-agents). [Lance Martin, "Context Engineering for Agents," June 23, 2025](https://rlancemartin.github.io/2025/06/23/context_engineering/). [Chroma Research, "Context Rot: How Increasing Input Tokens Impacts LLM Performance," 2025](https://www.trychroma.com/research/context-rot). [Microsoft Research, "LLMLingua-2: Data Distillation for Efficient and Faithful Task-Agnostic Prompt Compression," 2024](https://arxiv.org/abs/2403.12968).