The State of Context Engineering
dbreunig.comIt's nice that this article goes beyond the "context too long" scapegoat diagnostic.
There is more to it than just reducing content in context. LLMs attention mechanism works by finding graphs of related words and concepts and it works best when the relevant relationships are close together in the text. Length matters but how well the information is organized matters as well.
It's useful to be familiar with the concept of dependency length. The two sentences in this image ( https://benoitessiambre.com/img/dependencylength.png ) have the same words and same meaning but sentence C is organised to minimize the distance between related words.
This can also apply to paragraphs and sections. Ordering them so that related information is recursively closer together, helps AIs think better (also helps humans, Dependency Length Minimization, DLM is an idea from psycho-linguistics).
This is one reason progressive disclosure and sub agents are important. Including unnecessary information spreads apart surrounding information and makes it more difficult for AIs to figure things out.
For eg. optimizing skills, it can help to put more important information at the bottom of a skill. Usually after a skill is loaded, the AI will start “thinking”, talking to itself to try to reason through a problem. The thinking block will have an easier time considering the text closest to it, the last part of the skill, than information farther up.
Thinking blocks can produce a lot of text (a lot of it often hidden from users) that also spreads information apart. Sometimes, when an early instruction is still relevant but there’s been a lot of thinking and discussion added to the context since, it’s worth repeating a summary the early information ( keeping in mind that too many reminders can also spread other information apart ). I think however, LLMs are tuned to overweight very early information so that they don't disregard system prompts. You might be able to exploit this by placing some important facts near the very beginning of a conversation.
If you've ever written compression algorithms where repeated blocks are referenced instead of encoded, you'll be familiar with how related information being clustered allows for better compression. It's another example of the parallels between intelligence and compression and it hints at the information theoretic aspect of DLM.
Aren't these 2 parts in contradiction then?For eg. optimizing skills, it can help to put more important information at the bottom of a skill. LLMs are tuned to overweight very early information so that they don't disregard system prompts. You might be able to exploit this by placing some important facts near the very beginning of a conversation.https://en.wikipedia.org/wiki/Serial-position_effect
This is also how human memory works
My impression is that there's only a small area at the beginning of the context window that is artificially boosted, after that DLM becomes important.
Full disclosure: this is info I got from Claude. I was discussing DLM ideas with it a while back and while it mostly confirmed (maybe sycophantically) my ideas, it also said that rather than a pure decaying distribution effect with distance, it was U shaped because of the small boost at the beginning of the context window.
Very enlightening comment, thank you!
I like the idea of VictorTaelin's UniiChat [1] which summarizes every message into a single line and on top of that a binary tree of lines combined from lower levels. The context is then a view of that tree with more detail in recent messages and less detail in historical messages, plus a `zoom` tool that enables the agent to navigate the tree and read messages.
An implementation of this idea based on the pi harness: https://github.com/jonaslsaa/pi-optchat
[1]: https://gist.github.com/VictorTaelin/91837951a5ce5b38f341ec1...
> As we focus more on memory, we also need to think about strategic forgetting. Over time, overly precise memories become conflicted and messy, and nobody has yet figured out how to handle this.
> I think that’s the context problem of tomorrow.
This passage is explains why humans have imperfect memory. Seems the perfect memory we wish LLMs had impedes creativity and is maladaptive.
Strategic forgetting helps put the non-deterministic machine to best use. LLMs don't need perfect memory; that's covered by the computer itself.
I like that this justifies the continuing relevance of context engineering - a seemingly dated term - in the face of trends like loop engineering.
actually, I think agents have a lot to design after Zettelkasten yet. It's the concept of having an append-only policy and gardening to consolidate knowledge. IMO we've seen some, but there is still so much potential
I use a lightweight ticket system. Planning creates very detailed tickets with acceptance criteria. Implementation finishes with updating any info relevant to future work on the tickets as comments. The planner searches for previous related work both to avoid past pitfalls but also to get any potentially relevant info into its context. Works well with subagents ramping up quickly.
Cool. Do you find it useful for human review too, or is this mainly about the agents having the right abstractions to work with?
Just for agents. I track bigger picture work items in a normal ticket tracker (Linear, etc.) and the lightweight one (I use kata) is just for coordinating work and memory between agents.
> For me, around 200,000 or 300,000 is where I start looking. I’ll check, see it’s at 250,000, and think: I’ve got a big job coming up. I’m going to reset and start fresh
I don't find this to be true, at least with the newer models. I also find that the models fill up to 250k very quickly (presumably mostly by looking through the codebase). Without that context I find the models perform badly. I also don't really notice that large of a change until around 700-800k context and even then it isn't as bad as it used to be. I suspect, that (especially for coding), the long horizon RLVR tasks have helped with this quite a bit.
having an index of the code that the agent can use to find what its looking for instead of grep seems to work wonders
That crucially depends on what the index achieves. What do you aim for? Just lookup of symbols? Architectural information? Data flow systems?
For me, it's about getting the most work out of the €18 anthropic plan as possible. That means avoiding letting the context go over 300k or so.
I hate that they claim the pricing like that. It's 22.5e, we show our prices VAT included, not this misleading american bs
> The tea task for FireRed was in its weights, so it poisoned its context.
Mmm, to my mind, isn't this mixing completely different things? I thought poisoned context is stuff in the context that makes the model behave poorly. But if it's in the weights, that's just... model issues.
Yeah I thought the same here
The idea to get tea got into its context, and once there the model focused on that (rather than questioning if tea was the right drink)
Anecdotally (sample size == 1), I have observed that as my context grows longer, the model seems to get dumber. I'm not an advanced user with harnesses, etc.; I just use the chat interface, as that's all I can afford, but after a few turns the model (I'm talking frontier models here from OAI and Ant) seems to just do stupider things.
DeepSeek v4.1 flash seems to do this thing where it starts saying that we should wrap up the current work and start again "next session" (whatever that means), that the "context budget" is almost up, or some other wording I'm forgetting, as the context gets longer.
I've taken to just manually making it write a handoff file and clearing context completely. I'm pretty much manually compacting instead of my harness doing it. Feels weird, but it does work.
I think it’s trained into them so they emit a stop token with enough window left to generate a compaction summary
I would assume it's the harness sending the model a notification toward the end. At least Claude Code is designed to attach reminders with tool call output as a way for the harness itself to steer the model slightly.
Relatively sure it's not. I'm using pi and I can't find any place it's doing any such thing. My current theory is that it's behavior trained into the model either accidentally or intentionally, probably during post-training in some harness that _does_ give hints of that sort?
Any idea what the ~rule is or how it's meant to work? I assume it'd work better if I can wire up my harness to handle it, but no idea. It didn't seem to know itself, from it's training. I didn't search too much yet though.
You could also increase compaction.reserveTokens in ~/.pi/agent/settings.json to make pi compact more often. Also you can use /compact <instructions> to specify what facts should be included in the summary.
One hard part is that from a handoff file you only have information for 1 session, so you have to manually determine next week which facts are still true. At Substrate we solve this problem using our memory: we have 1 memory outside of an agent and track the source of every fact. https://trysubstrate.co/l/a2hn
I've also found, as the model context gets larger, my work to supervise increases.
Like a see-saw - we just need that balance, man!!!
Let's bump that sample size to 2. I've observed the same and that observation has heavily influenced the design of my coding agent harness.
3
I have auto-compact on at 400k
4
I try to manually manage as much of the context as possible. Most of the time, I limit myself to a context window of 200k, and it works great for my use cases.
I can't cite the sources at the moment but this is already well known - a few papers on it and Anthropic say somewhere in their docs that you want to get across the minimum amount of context required to achieve your goal. Quality diminishes the more you dump in.
That calls lost-in-middle issue with LLM models. You have to use appropriate workflow patterns like map/reduce to handle big context. But be sure to run experiments on dataset first to understand what logic works well and stable. Where NOT! And how it can be combined to reduce LLM stohastic problems in prod
It does, and, when combined with Claude's stingy cache TTL of 5 minutes (unless you up it to 1h at a sizeable increase in token cost) for API billing, means that some sessions cost like 10x what they should because of cold caches.
> When building intelligent applications, harnesses, and workflows, our job is to choose when to be deterministic (what to put in code) and when to be probabilistic (when to let the model drive).
I really like this final sentence from the presentation. When I read this phrase, I realized that it captured one of the core dynamics that I've been using to figure out when and how to apply AI to a task. I often find myself trying to figure out if it is better to A) ask the AI to do a task for me or B) ask the AI to build something that can do the task for me. A major component of that answer comes down to how deterministic or probabilistic the task is.
I like the overall idea of deliberate forgetting. And I found checkpointing into artifacts (code, design, intent) to start with fresh context to work well for that purpose.
Interestingly, the artifacts include not just code but alao retroactive design and retroactive “execution plan”. I feed them to another agent to validate the approach and resume work with fresh context and update mission statement.
Unexpectedly, design doc and an execution plan are now outcomes of coding process, not inputs to it. I’m not sure how to think about this.
You say you feed them to another agent to validate your approach, is this because you feel that a fresh context is necessary for a less biased judgement?
> Several times, Gemini 2.5 became convinced it needed to get tea, even though it
> was playing the older game. The tea task for FireRed was in its weights, so it
> poisoned its context. It spent hours trying to find or give the guard tea. Once
> it had been added to its to-do list, the model could not get past it.
The information to get tea was in the weights (and a mistake, it should have pulled lemonade).
Once pulled it added "iced tea" to its to-do list, which gets put into context. The model then couldn't escape the idea.
"iced tea" in context was a symptom - it might influence the downstream - but it is hard to establish that. The root problem was in weights - naming it 'context poisoning' is poisoning our reasoning context :)
You are confusing two different things.
It hallucinated that it needed tea.
Once that hallucination got into context, the context was poisoned.
If you don't want to believe that, then I cannot force you to, you can believe whatever you want.
But I (and the author) am explaining an otherwise well known problem with hallucinations getting into context.
My point is that calling it context poisoning is confusing because of two things. First - it is hard to establish how causal it is for the downstream. Second - cleaning the context would not fix it - because it would be poisoned on next round - you need active measures to fight it.
It is also a bit tongue in cheek - if you did not notice - because of course all of this can be explained if needed even if the name alone suggests something else.
You are still confused here.
On the next round, it's likely to correctly determine lemonade.
Hallucinations are usually outliers. So 1 in 10 times it gets confused and picks tea. Those 1 in 10 times end up with poisoned context.
The models are not deterministic by design. Identical inputs produce different outputs. So if you roll a bad output, it can poison your context. Flushing the context can easily fix the problem (unless like the in the pokemon harness, you roll over a brief of the previous context)
Very informative article but why this term coining? The phenomena and strategies aren't specific to LLMs but are well known from other areas of computer science (and psychology). For example, I would have called "offloading" just "swapping", and "confusion" just "pollution".
This is why I think the next evolution will be giving models full control of their own context, where they can edit it just like a file. This idea was in a recent paper, where models could even be specifically trained for context management [1]. For the problem of the prefix cache being invalidated, they thought of implementing a suffix cache
So like- I considered this idea of using a fast decision model to prune before compaction, but I scrapped it because A; im not bottlenecked by speed, and B; cache invalidation dark magic
From my understanding, if we have turns [A, B, C, D, E] and we decide that turn C is useless context, then we lose our cache for the [D, E] segment. (plz correct me if im wrong, I havent tested this)
Personally, I pay no mind to session state nowadays. I exclusively use provider native hosted compaction, which if you believe the providers, is the way to go. For context management- I spend most of my effort on the things which dump output back to the model.
If anyone has any good reads on benchmarking OpenAI and Anthropic hosted/native compaction against custom stuff, I would love to read!
Yeah, I also think decision models for pruning context isn't worthwhile. You're paying to load it into context (of your decision model), in order to save on loading the same data into context (of your generation model). The only way the economics makes sense is if you need a much smarter model for the compaction, but can't pay for it to load all of that context (because it's uncached or something). That sounds like a niche but it really doesn't exist. Cheap models are good enough, and smart models are cheap enough.
Personally, I use a harness extension that auto-compacts my context just before the cache expires. Combined with the default compact-when-full, this ends up being pretty much always what I want. It's one less thing to think about.
It's the "way to go" for them in particular because they hide their reasoning traces, and the compaction has to account for them to be effective, so they have to do it internally, one-size-fits-all. It's not a problem with open models which can have different algorithms for different purposes, and/or any competition.
It's because the cache comes from the prefix. `f(f(f(A,B),C),D)` is quite different from `f(f(A,B),D)`.
Yes, that's my understanding- though I realize that OpenAI has made changes to what you can do with caching over the API recently, so you may be able to do some fancy stuff to get around this.
I use dynamic context compaction in open code plugin. Then modified it further to include eviction.
Since I run everything local, speed benefits from keeping context low ~128k. This gets me through a single agent on multiple tasks through the same repo. It seems to break around 6M but that's more probablistic.
KV cache is definitely valuable but context poisoning happens regardless of size and it's probability increases regardless of intelligence. So at some point you're going to suffer from context overload regardless.
Until a few months ago I did it this way, but OpenAI and Anthropic built their own harnesses, and since those end up conflicting with the harness I built myself, having no setup at all actually ended up performing better. As a result, it doesn't seem necessary to set up rules or memory separately, and if I do, using skills or agents made of only logging and flow (without things like rules or guardrails) seems like the best option going forward.
I think that makes sense for now in individual contributor cases, but the turnkey solutions have gaps when it comes to the most powerful capabilities and I don't see harnesses going away any time soon for that.
E.g. you can't use coworking/projects for shared memory in most domains because this hinges on compacting/refactoring true reusable facts and organisational knowledge (as opposed to code, schemas, technical references). The heuristics to find and decide on candidates for memory in this case are quite specific to the institution you work in. For example, the boundaries of the domain of "sales" versus "marketing" vary lots from place to place, as does the trustworthiness of certain staff, the way they they communicate or make decisions that contain the knowledge you want to store to memory, and so on.
I'm finding harnesses are still vastly better for grabbing this sort of complex institutional knowledge. Also, they're just a bit of a safety net for any frontier labs going bankrupt in the next couple of years, if we need an easy way to port over to open weights.
Is it just me or others also feel like “no respect” for this kind of knowledge? Before I was proud to know about TCP connection protocols, 2FC, parsers, compilers… and much of that knowledge had applications on the software I had to build.
But nowadays I have very little interest in knowing how “to prompt better” or why after around 100K tokens of context things start to suck for agents, or compiling tons of MD as “skills”. I don’t find that stuff challenging or interesting (granted, I don’t mean that I know all about it) and I have the feeling that it’s not like learning about operating systems or relational dbs.
I think part of my overall rejection to LLMs comes from this (of course the actual knowledge of how LLMs work under the hood is super interesting, but not their usage Id say)
I think it's definitely not as "cool", but it's almost a different problem class entirely since there's no correct answers here.
Knowing TCP connection protocols or deep compiler infra is impressive insofar as it's a mountain of knowledge that you've conquered and can stand on with a nice view, whereas "context engineering" and current AI engineering is a murky domain with no correct answers because... nobody alive knows the correct answers.
There's something different about wading forward into unexplored territory, or being able to establish techniques that nobody has attempted before. A lot of it will be noise or garbage (especially given a lot of it is LLM code), but I'd be careful to not mistake clever greenfield engineering with garbage simply because there's not a book written about it you can use as an oracle.
>This is the advice I keep giving now. Someone says, “My coding agent’s doing something weird. Claude’s been nerfed.” And I ask, “How big is your context?”
That’s really true, but on the other hand, if there’s too little information, it also ends up working in strange ways. Most AGENT code anti-patterns work well at a medium amount of context, but when there’s too much or too little information, they don’t behave properly.
Forgetting is the part I see break first. Old facts in memory don't get deleted, they get contradicted, and the model happily picks whichever one is closer in the prompt. What's worked better for me is having this really cool way of jev compaction, there was a youtube video out there somewhere that spoke about how to get compaction to be super fast, cheap and have it retain the most useful information from the context compaction.
I can't tell if this engineering or just autocomplete that is mostly helpful, sometimes.
Stupid question that I could look up myself but asking here…
Is context represented in by the original text, or translated into token indexes? Because if it’s in the original form, it seems obvious that either the server should be compressing into token indexes, or on handshake the server gives the client its dictionary and on request it translates it
The cardinality of terms feels too high and too much semantic overlap between different terms.
I would like to counter with Theo Browne's "Your Prompts Kinda Suck" [1] I'm not a "fan" of Theo but his perspective here is useful.
Anecdotally I only stuggle with this with Claude Code. Codex has been great at auto managing its context and, when combined with how I handle memory outside of the session, I haven't thought about context window size in ages when using Codex.
I thought the price reduction was because both companies implemented Deepseek's published research which demonstrate effective approaches to lowering the cost of inference while maintaining quality. Thanks to Deepseek, they did lower their costs and they passed that in to their customers.
interesting read, thanks for sharing
"Engineering"