Agentic AIContext EngineeringAgent MemoryLLMMCPRAGSoftware ArchitectureReliability

Context Engineering in Production: Beyond Agent Memory

7 August 2026

The discipline above agent memory: what actually loads into the context window each turn - compaction, note-taking, JIT retrieval, and why two teams disagree on sub-agent isolation.

If you read my Agent Memory in 2026 post, you already know how to pick between Mem0, Zep, Letta, and LangMem for long-term state. That solves persistence. It does not solve the problem that shows up the moment your agent actually runs: on any given turn, what exactly goes into the context window, in what order, how much of it, and why?

That question has its own name now, and it is not a rebrand of prompt engineering. Context engineering is the discipline of deciding, turn by turn, what the model actually sees. It split off from prompt engineering for a structural reason. A chatbot gets one prompt per turn. An agent generates a growing pile of tool calls, observations, retrieved memories, and partial plans, and all of it competes for the same finite window. Dex Horthy, who coined the term while writing 12-Factor Agents, named the failure mode the “dumb zone” - the middle 40-60% of a large context window where recall degrades and instructions get lost even though the tokens are technically present.

This post is for architects who have already solved memory, or are about to using the frameworks from my last post, and are now hitting the next wall: agents that retrieve the right facts and still degrade, hallucinate tool calls, or burn ten times the tokens they should. I will walk through the techniques Anthropic, Manus, and the wider field have converged on for managing context as a finite resource. I will show where two credible engineering teams reached opposite conclusions about sub-agent isolation in 2025, how that fight actually resolved by 2026, and get concrete about what it costs when nobody owns this problem.

Table of Contents
Why context engineering became unavoidable
Context engineering is not agent memory
Context as an operating system: five quality criteria
Compaction: summarize before you run out of room
Structured note-taking: memory that lives outside the window
Just-in-time retrieval: load nothing until it earns its place
Isolate or share: how the sub-agent fight got resolved
Design around the KV-cache
What it costs when nobody owns this
A decision framework
Closing
Sources

Why context engineering became unavoidable

Every token in a transformer’s context attends to every other token, which is why an n-token context costs roughly n² worth of pairwise relationships to process (Anthropic, September 2025). Models also see far more short sequences than long ones during training, so they have less practice with context-wide dependencies at the far end of a large window. The result is not a hard cliff. It is a gradient: models stay capable at long context but get measurably less precise at retrieval and long-range reasoning as the window fills.

Chroma’s Context Rot study (July 2025) put numbers on this, testing 18 models including GPT-4.1, Claude 4, Gemini 2.5, and the Qwen3 family on simple retrieval and text-replication tasks, the kind a model should ace regardless of input length. Reliability degraded well before any model’s documented context limit, and non-uniformly: performance depended on where in the window the needed fact sat, how similar it was to surrounding distractors, and how the text around it was structured, with double-digit-percentage accuracy drops in several of the harder configurations. The finding to remember: models do not use a 200K-token window as 200K tokens of equal-quality attention. They use it as a shrinking budget.

Horthy’s “dumb zone” (see 12-Factor Agents above) is the practical version of the same phenomenon: the middle 40-60% of a long context where recall degrades even though the tokens are present and correct. “Own your context window” is one of his twelve factors, and it is the reason the term “context engineering” exists at all rather than just being an extension of prompt engineering.

This is why “wait for a bigger context window” has been the wrong answer for two years running. Bigger windows postpone the point where you run out of room. They do not fix the fact that every additional token, cached or not, relevant or not, taxes the model’s attention budget. That is what makes curating the window a discipline instead of a workaround.

Context engineering is not agent memory

It is easy to conflate the two, and a lot of 2026 vendor content does exactly that. My Agent Memory post covered Mem0, Zep/Graphiti, Letta, and LangMem - all of them answer “how does my agent persist and retrieve facts across sessions?” That is memory. It is a storage and retrieval problem.

Context engineering sits on top and answers a different question: on this turn, right now, what actually gets loaded into the window? A memory search might return twelve candidate facts. A RAG pipeline might return eight document chunks. The system prompt, the tool definitions, the last six turns of conversation, and a scratchpad file are all also competing for the same budget. Deciding what subset survives, in what order, and how compressed, is context engineering’s job. Memory is one input into that decision, not a synonym for it.

Anthropic frames this precisely: context is the full set of tokens sampled at inference, which includes system instructions, tool definitions, MCP-supplied data, message history, and retrieved memory alike, and context engineering is curating that entire set on every inference call, not just the retrieval step (Anthropic). A great memory architecture with a sloppy context-assembly step still produces an agent that drowns in its own retrieved facts.

Context as an operating system: five quality criteria

A March 2026 arXiv paper by Vera Vishnyakova, Context Engineering: From Prompts to Corporate Multi-Agent Architecture, proposes five quality criteria for treating context as an agent’s operating system:

Criterion	What it asks	Fails as
Relevance	Does the available information match what the agent needs to decide?	Irrelevant tool results and stale memory padding the window
Sufficiency	Is everything needed for a correct decision actually present?	The agent guesses because a required fact never made it in
Isolation	Does each agent or sub-agent see only what it is supposed to see?	Cross-contamination between sub-agents, or context leaking across tenants
Economy	Is the context the minimal size needed, with minimal recomputation?	Ballooning token cost, KV-cache invalidation, slow responses
Provenance	Can every piece of context be traced back to a verifiable source?	The agent can’t say why it believes something, or acts on unverifiable input

Heads up: this is a single-author preprint (submitted 10 Mar 2026, arXiv:2603.09619), not a peer-reviewed or widely cited standard. Treat the five criteria as a useful checklist, not an industry consensus - the paper is more useful as vocabulary than as authority. Its enterprise data points are worth knowing regardless of what you think of the framing: 75% of organizations plan agentic AI deployment within two years, per the Deloitte 2026 survey the paper cites, while deployment has been “surging and retreating” as organizations hit scaling complexity, per the KPMG 2026 research it also cites.

The paper’s example is Klarna, and it is worth getting the underlying story right rather than the compressed version. Klarna’s AI customer service assistant handled 2.3 million conversations in its first month, two-thirds of all service chats, doing the equivalent work of 700 full-time agents, cutting resolution time from 11 minutes to under 2, and dropping repeat inquiries by 25% (Klarna’s official announcement, February 2024). By May 2025, CEO Sebastian Siemiatkowski told Bloomberg that cost had been too dominant a factor in how the system was built, and Klarna began rehiring human agents after customer satisfaction dropped (Forbes, May 2025). Whether you call that an isolation failure, a sufficiency failure, or simply an under-invested support model is a matter of interpretation. What is not in dispute is that a system that looked great on aggregate metrics in month one needed a costly course correction fourteen months later.

Compaction: summarize before you run out of room

Compaction takes a conversation nearing the context limit, summarizes it, and restarts a new window from that summary instead of the raw history. Anthropic first described this as an internal Claude Code behavior in September 2025: summarize message history while preserving architectural decisions, unresolved bugs, and implementation details, discard redundant tool outputs, and continue with the compressed summary plus the five most recently accessed files.

That description has since become a real, versioned API feature rather than just an internal technique. As of the compact-2026-01-12 beta, server-side compaction is a first-class option on the Messages API: you enable it with a compact_20260112 edit, it fires automatically once input tokens cross a configurable threshold (150,000 by default, minimum 50,000), and you can supply custom summarization instructions or pause immediately after the summary to splice in verbatim recent messages before continuing (Claude Platform Docs, “Compaction”). It is currently in beta on Claude Opus 4.6 through 5, Claude Sonnet 4.6 and 5, and Claude Fable 5 and Mythos 5.

import anthropic

client = anthropic.Anthropic()
messages = [{"role": "user", "content": "Help me build a web scraper"}]

response = client.beta.messages.create(
    betas=["compact-2026-01-12"],
    model="claude-opus-5",
    max_tokens=4096,
    messages=messages,
    context_management={
        "edits": [
            {
                "type": "compact_20260112",
                "trigger": {"type": "input_tokens", "value": 100_000},
                # Custom instructions replace the default summary prompt entirely.
                "instructions": (
                    "Preserve architectural decisions, unresolved bugs, and "
                    "implementation details. Discard redundant tool outputs."
                ),
            }
        ]
    },
)
# The response may include a `compaction` content block. Append the full
# response to your message list and keep going - the API drops everything
# before the compaction block on the next call.
messages.append({"role": "assistant", "content": response.content})

The hard part is still the tuning, not the mechanism. Anthropic’s guidance is to optimize the summarization prompt for recall first on real, complex traces, then iterate for precision by cutting superfluous content, since overly aggressive compaction quietly drops context whose importance only becomes obvious later. A lighter-touch alternative now ships alongside it as a separate, named strategy: clear_tool_uses_20250919 under the context-editing API removes only old tool results (optionally the tool calls too) once a token threshold is crossed, without summarizing anything, and pairs with a clear_thinking_20251015 strategy for trimming extended-thinking blocks (Claude Platform Docs, “Context editing”). Reach for tool-result clearing first; it is cheaper and safer than a full summarization pass, and only escalate to compaction once clearing alone cannot keep you under the limit.

Trade-off: compaction preserves conversational flow well for tasks with extensive back-and-forth, but every summarization pass is itself a lossy operation performed by a model that cannot know which detail will matter in step forty. The docs are explicit that it is a poor fit for tasks needing precise recall of early conversation details, or workflows that lean heavily on server-side tools, where token accounting gets muddied by internal retries. Tune it, do not fire-and-forget it.

Structured note-taking: memory that lives outside the window

Structured note-taking, or agentic memory, is the agent writing notes to a file outside the context window and reading them back in later. Anthropic described this in September 2025 as a NOTES.md file or an evolving to-do list that survives a compaction or context reset, and it now ships as a named, documented tool (memory_20250818) that pairs directly with context editing: when clearing or compaction is about to remove a tool result, Claude gets an automatic warning and can write the important part to a memory file first (Claude Platform Docs, “Context editing”). Manus’s version, the todo.md file it rewrites at every step of a long task, does the same job but frames it as attention management rather than persistence: rewriting the plan and pushing it back toward the end of the context “recites” the objective into the model’s recent attention span, which counters lost-in-the-middle drift on tasks that average around 50 tool calls (Yichao “Peak” Ji, Manus, 2025).

from pathlib import Path

NOTES = Path("notes.md")

def update_notes(entry: str) -> None:
    # Append-only, mirroring the KV-cache stability rule below - a file that
    # gets rewritten from scratch every turn is as bad for caching as a
    # context window that does.
    with NOTES.open("a") as f:
        f.write(f"- {entry}\n")

def recite_objective(goal: str, history_tail: list[dict]) -> list[dict]:
    # Push the goal toward the end of context every turn - Manus's todo.md
    # rewrite does exactly this, and it requires no architecture change.
    return [*history_tail, {"role": "system", "content": f"Current objective: {goal}"}]

This is deliberately lighter infrastructure than Zep’s temporal graph or Letta’s tiered memory service from my last post. The distinction that matters here is not the storage mechanism, it is that note-taking is a context-engineering decision about when a fact re-enters the window, not a memory-architecture decision about how it is indexed. Nothing forces a salient fact into the notes the way Zep’s extraction pipeline forces one into the graph; the agent has to choose to write it down.

Trade-off: minimal overhead, but the technique is only as good as what the agent decides is worth writing. Pair it with an explicit prompt instruction about what belongs in the notes, or you get a to-do list that tracks task completion but not the reasoning behind decisions.

Just-in-time retrieval: load nothing until it earns its place

Rather than pre-loading everything that might be relevant, an agent can maintain lightweight references, file paths, query strings, links, and load the actual data only when a tool call needs it. Claude Code runs a hybrid of this: CLAUDE.md files load upfront because they are cheap and stable, while glob and grep retrieve files just-in-time, sidestepping stale indexes and the overhead of maintaining a syntax tree (Anthropic).

Claude Agent Skills make the same pattern explicit as a three-level protocol: a skill’s name and description (roughly 100 tokens) sit in context at startup regardless of whether it is used; the full SKILL.md body (kept under 5,000 tokens by convention) loads only once a request matches that description; and any additional files or scripts the skill references load only when the task actually needs them, with script code never entering context at all, only its output does.

---
name: pdf-form-filling
description: Fill and flatten PDF forms. Use when the user needs a PDF form completed.
---
# Only this frontmatter, a few dozen tokens, sits in context at startup.
# The instructions below it, and any linked scripts or reference docs,
# load only when a request actually matches the description above.

The trade-off Anthropic is explicit about: runtime exploration is slower than retrieving pre-computed data, and it only works well when the agent has good navigation heuristics, naming conventions, folder hierarchy, timestamps, to decide what to fetch. Drop an agent into a directory of files named data1.json through data400.json and just-in-time retrieval has nothing to navigate by. The hybrid model, some data pre-loaded for speed, the rest fetched on demand, is usually the right default rather than committing fully to either extreme.

A newer variant worth watching rather than adopting outright: internal developer platform vendor Port.io has been pushing what it calls a “context lake”, pre-joining frequently traversed relationships (which service pages which on-call engineer, which repo backs which ticket) into the record itself at write time, so an agent never spends tokens re-deriving them at read time (Port.io, “Context lake”). Port reports 56-80% token reductions depending on configuration. Treat that figure the way you should treat any vendor-reported benchmark in this post: it is Port’s own number, on Port’s own product, and the underlying idea, compute relationships once at write time instead of on every agent turn, is sound engineering independent of whose product implements it.

This looked, for most of 2025, like two credible teams reaching opposite conclusions on the same problem. By 2026 it is closer to a settled default with a clear exception, and it is worth walking through how that happened rather than just quoting the ending.

Anthropic’s multi-agent research system runs a lead agent that delegates to specialized sub-agents, each exploring with its own clean context window, often tens of thousands of tokens, and returning only a condensed summary (typically 1,000-2,000 tokens) back to the lead agent. This produced a substantial improvement over single-agent systems on complex research tasks, because the detailed exploration stays isolated while the lead agent focuses on synthesis, at a real cost: Anthropic’s own accounting put multi-agent token consumption at roughly 15 times a single chat turn, against roughly 4 times for a single tool-using agent, and the architecture only pays for itself when the task’s value clears that multiplier (Anthropic, “How we built our multi-agent research system,” June 2025).

Cognition’s Walden Yan argued the opposite in “Don’t Build Multi-Agents” (June 2025), written while building Devin: sub-agents with isolated context routinely produce conflicting, unusable results, because they lack shared awareness of each other’s decisions. His illustrative failure asks for a Flappy Bird clone split into two sub-tasks, a background and a bird; the background sub-agent mistakes its brief and builds a Super Mario-style scene, the bird sub-agent builds something that looks nothing like a game asset, and neither could see what the other had already decided. Yan’s fix at the time was the opposite of isolation: share complete agent traces between agents, and default to a single-threaded linear agent unless you have a specific reason not to.

Nine months later, Cognition shipped the architecture it had argued against. “Devin can now Manage Devins” (March 2026) gives the main Devin session a coordinator role: it scopes a large task, delegates scoped pieces to managed Devins that each run in their own isolated VM with their own terminal and browser, monitors their trajectories, and compiles the results. Cognition’s own justification for the reversal is Anthropic’s original argument, restated almost word for word: “when one agent tries to handle too many things in a single session, context accumulates, focus degrades, and the quality of each subtask suffers.” The essay was never formally retracted, but the product now does the thing it warned against, with the caveat that made it work: isolate the sub-agents, but keep the coordinator’s decomposition explicit enough that the pieces do not collide the way the Flappy Bird example did. Cognition’s own suggested use cases for managed Devins are telling: QA every page of an application in parallel, audit every service in a codebase, refactor independent batches of components, one managed Devin per unit of work that does not need to see what the others are doing.

I could not find a rigorously sourced industry-wide count of how many production deployments use this pattern specifically, and treat any number you see quoted for that (including ones with a specific percentage attached) with the same skepticism this post has applied to other self-reported figures. What I can verify directly is the convergence between these two specific teams: the vendor that argued most forcefully for isolation (Anthropic) and the vendor that argued most forcefully against it nine months earlier (Cognition) now ship the same shape, one orchestrator holding full context and spawning isolated workers that return compressed summaries. Multiple secondary reports claim OpenAI, Microsoft’s Agent Framework, and LangChain have converged on the same pattern; I was not able to confirm that directly against each vendor’s own current documentation, so take it as directionally credible rather than verified. The Anthropic-Cognition disagreement itself did not resolve toward one side, it resolved toward a decomposability test that both original arguments were implicitly making: if the sub-tasks are genuinely independent and only the final result matters, as with parallel QA runs or per-service audits, isolate and summarize. If a decision made in one branch changes what the correct decision is in another, as with a background and a foreground character in the same game, isolation reproduces the Flappy Bird problem regardless of which vendor’s framework you use. Given the token multiplier both teams now agree on, run that test before reaching for isolation by default, not after something breaks.

Design around the KV-cache

If you run agents at any real volume, the metric that moves the numbers most is not a quality metric at all, it is the KV-cache hit rate. Manus reports an average input-to-output token ratio of around 100:1 in its agent loop, which is normal for tool-using agents and very different from a chatbot’s ratio. Every cache miss reprocesses the full growing prefix, and the discount for a hit is not small and not specific to one vendor’s snapshot in time: as of August 2026, Anthropic prices a prompt-cache read at a flat 0.1x the base input rate across every current Claude model, which works out to $0.30 per million tokens against a $3 per million token base rate for Claude Sonnet 5 at standard pricing (Claude Platform Docs, “Pricing”). That is the same 10x ratio Manus reported for Claude Sonnet back in 2025, which is worth noting on its own: the discount has held steady in relative terms across at least two model generations, so it is a durable planning assumption, not a number that will quietly move under you.

Three practices protect the hit rate. Keep the prompt prefix stable: a timestamp precise to the second at the top of a system prompt silently kills the cache from that point forward. Make context append-only: never edit a past action or observation, and make sure your JSON serialization has deterministic key ordering, since some language runtimes do not guarantee that by default. And where a provider requires manual cache breakpoints rather than automatic incremental caching, place them deliberately, at minimum at the end of the system prompt.

The subtler rule is mask, don’t remove. As an agent’s tool count grows, RAG-style dynamic loading of tools looks tempting, but adding or removing tool definitions mid-run invalidates the cache (tool definitions typically sit near the front of the serialized context) and confuses the model when earlier turns reference a tool that is no longer defined. Manus’s alternative is to keep the full tool set in context and mask which tools are selectable at decode time via prefill, using the Hermes function-calling convention as an example: an unconstrained “auto” prefill, a “required” prefill that forces some tool call, and a “specified” prefill that forces a call from a named subset.

# Illustrative - mirrors Manus's approach with the Hermes function-calling
# format. Constrain the action space via prefill instead of editing tool
# definitions, so the cached prefix never changes mid-run.

PREFILL_MODES = {
    "auto": "<|im_start|>assistant",
    "required": "<|im_start|>assistant<tool_call>",
    "specified": '<|im_start|>assistant<tool_call>{"name": "browser_',
}

Consistent tool-name prefixes (browser_, shell_) make this practical without a stateful logits processor: masking to “anything starting with browser_” is a string match, not a lookup table.

Trade-off: none of this matters for a low-volume internal tool where a few extra hundred milliseconds and a few extra cents are noise. It matters enormously at agent-loop volumes, where a 100:1 token ratio means the prefill cost, not the generation cost, dominates your bill.

What it costs when nobody owns this

Context rot is usually framed as a quality problem. It is also, directly, a cost problem, and the two are the same root cause viewed from different angles. EY’s own analysis of agentic AI token costs found that a simple linear workflow (retrieve, respond) cost around $0.04 per interaction in 2023, while an orchestrated 2026 agent using tools, MCP servers, reasoning steps, sub-agents, and retries costs closer to $1.20 per interaction, roughly 30 times more (EY, “Agentic AI Enterprise Token Cost”). None of that increase is inherently wasteful; orchestration does more work. But a meaningful fraction of it is context that was never curated: retries caused by a model that lost the thread, redundant tool calls because context wasn’t isolated cleanly between sub-agents, cache misses from an unstable prefix.

Klarna’s reversal, described above, is often read purely as a quality story: the assistant got worse, so humans came back. Siemiatkowski’s own framing to Bloomberg was that cost was too dominant a factor in the design. Read alongside the rest of this post, that is not two separate failures, quality and cost, it is one failure showing up on two dashboards. A system built to minimize token spend without curating what actually went into each decision will be cheap and unreliable at the same time, because the same missing discipline, deciding what the model actually needs to see, is what would have fixed both.

A decision framework

Match the technique to the shape of the problem, not to what is trending.

If context is dominated by a single long-running conversation or coding session, compaction is the first lever. Tune the summarization prompt on real traces, maximize recall before you trim for precision.
If the agent needs to track progress or objectives across dozens of tool calls without derailing, structured note-taking (a todo.md or NOTES.md) is cheaper than compaction and works well alongside it.
If you have more tools, documents, or reference material than could ever justify sitting in context at once, invest in just-in-time retrieval and the navigation signals (naming, folder structure, metadata) that make it work.
If a task decomposes into independently verifiable pieces where only the final result matters, isolate sub-agents and return condensed summaries; this is now the default across Anthropic, Cognition, OpenAI, and LangChain for a reason.
If the task involves interdependent decisions where one step constrains the next, keep it single-threaded or share full traces between agents; isolation reproduces Cognition’s Flappy Bird problem regardless of how mature the tooling is.
Either way, remember the multiplier: a multi-agent architecture costs roughly 15x a single chat turn in tokens. Isolate because the task decomposes cleanly, not because parallelism looks impressive in a demo.
Regardless of task shape, once you are past prototype volume, design for KV-cache stability. Stable prefixes and append-only context are close to free relative to the latency and cost they save.

Closing

The question worth asking is not “which memory framework did I pick.” It is “on this turn, does the model see the smallest set of tokens that gets the decision right.” Memory answers what facts exist. Context engineering answers what actually gets used, and it turns out that getting that wrong is expensive in tokens and in trust at the same time, not one or the other.

The field is young enough that the ground is still moving, and it moved visibly while researching this post: Horthy named the discipline in 2025, Anthropic formalized it in September that year, Cognition reversed a widely-cited architectural position by March 2026, and compaction went from a described Claude Code behavior to a versioned, parameterized API beta in the same window. Treat everything here as the current best practice, not a settled standard, and check the primary docs before you build on any specific number or API name in this post, since the pace so far suggests at least some of it will move again before this time next year.

Sources

Papers

Vendor architecture posts and current docs

Independent surveys and reporting

Related reading on this blog

Discussion

Reader comments

Loading comments…