The Memory Problem
Press enter or click to view image in full size
Remember the movie Memento? The plot follows a man with radical short-term memory loss who uses tattoos and handwritten notes to track clues as he hunts down his wife’s killer. But he has this fragmented perception that leads him to question what’s real and whether he’s manipulating himself. The more I work with agents, the more it feel like they are struggling with the same problem.
Maybe you’re coding something or asking your agent to perform some operations for you and its context window begins to fill up with erroneous details. It goes off on a tangent and seems to have completely lost the thread of whatever you were working on.
It’s been said that if you start an intern and an agent in the same role, on day one the agent will be better. But as time goes on, the intern will build much-needed context about the way the organization works. They will learn undocumented patterns, cultural cues, all the kinds of things that a large language model will not have in its training data or prompts.
A large language model is not like a traditional piece of software. It’s a snapshot frozen in time, a compressed statistical representation of patterns in data at the date of training. We address this unfrozen caveman syndrome by giving the model access to new data through tools like web search, filesystem reads and mcp. This is great, except it is wildly unsupervised and very prone to filling up that precious context window with clutter and irrelevant information.
What is Agent Memory?
It’s the things you want your agent to know and store and be consistent about all the time. It’s the stored, managed, and updated context about the work you’re doing.
A lot of times when you’re coding, you don’t notice the need for agent memory because the code base serves as that memory. Anytime you and your agent need to remember what it is you’re working on, you look at the code base and that’s the context.
But when you’re doing other types of tasks, that don’t necessarily produce specifically formatted text artifacts, context can get lost.
I got really into thinking about this when I started using OpenClaw on my home server. I was trying to get it to do things like configure Home Assistant dashboards or set up Z-Wave automations. It would constantly forget the things that felt like we had discussed and agreed were important. It was and continues to be very frustrating.
OpenClaw has an established memory pattern that makes use of vectors and markdown files and all the things we’ll talk about. But you find quickly that it doesn’t work well enough and you have to think about the memory architecture.
Many agents we use don’t have any notion of memory patterns, and we have to build them. CLI tools like Claude Code, Open Code or Code Puppy follow this pattern. You want to do all kinds of autonomous agentic stuff with them, but they don’t naturally remember things.
Types of Memory
Vectors
When you store something as a vector, you’re essentially just turning it into a list of numbers using what’s called an embedding model. for example, the sentences “I love dogs” and “dogs are amazing” would end up close together in vector space.
Once you’ve created those vectors, you can store them in a vector database or even in just a simple file like JSON or SQLite.
Once you’ve stored the vectors, you can then find them by closeness. Using a technique called cosine similarity, you can compare vectors and find the ones that are closest to the one you started with.
This allows us to do a technique called semantic matching in which we can look up records that are related to the terms we’re operating on.
This pattern forms the basis of RAG or Retrieval-Augmented Generation
In a RAG system, when a user asks a question, you first convert that query into a vector using the same embedding model. Then you search your vector store for the most similar vectors, which map to pieces of information that are semantically related to the query. Those retrieved snippets are then passed along with the original question to your language model, which uses them as context to generate a more accurate and grounded response.
Text files
A common pattern for giving agents memory is to provide text files for them to use with instructions. these are often referred to as context files. You’ve heard the expression “context engineering” partially in reference to this practice.
Agent instructions
This is the AGENTS.md file or other vendor specific instructions file that is always added into the context window at the beginning of every agent session. It’s very useful for overarching instruction or best practices. However, if you try to cover every case and the file gets too large, you run the risk of context window clutter and conflicting or irrelevant instructions to a particular task.
Skills
Agent skills aim to solve the granularity problem of agent instructions. A skill is a text markdown file and associated scripts or tools that are only invoked when the agent determines (via similarity or routing logic) that they are relevant to the task. For example, you may have a PDF skill which provides your agent instructions on managing a PDF document. However, you don’t want these instructions to be cluttering up your context window on non-PDF related tasks. Instead, the agent would match the prompt to the skill description and invoke the skill when you’re working on a PDF. This allows agents to “remember” things that we have taught them how to do along the way. And also, like Neo in the Matrix, to install learnings from other agents.
Use this skill whenever the user wants to do anything with PDF files. This includes reading or extracting text/tables from PDFs, combining or merging multiple PDFs into one, splitting PDFs apart, rotating pages, adding watermarks, creating new PDFs, filling PDF forms, encrypting/decrypting PDFs, extracting images, and OCR on scanned PDFs to make them searchable. If the user mentions a .pdf file or asks to produce one, use this skill.
Memory files
Oftentimes there is a lot of context that don’t fit neatly into agent instructions or skills. This may be research findings, conversation history, any other types of text that we generate with an agent that we want the agent to know about later. We can write these to memory files and ingest them into the context window, but this technique suffers from the same context clutter and relevancy problem that we discussed earlier. We can use a vector store to point us to contextually relevant documents and this can work reasonably well in many cases. In fact, this is similar to how systems like OpenClaw work.
Structured documents
We can also use traditional data stores as a source of memory for agents. even as something as simple as a SQLite database on the local file system can provide a very useful tool for managing relational pieces of information. Similarly, you can use other forms of structured document like JSON to store pieces of information that lend themselves to that structure. We can then easily define skills for querying these documents in a performant and useful way. Agents are good at that kind of thing.
Edge vs Cloud Storage.
Any of the above storage methods can be done using web services as well as local file systems. hosted vector DBs, context documents in GitHub, skills repositories, or databases as sservices. All allow you to provide the same memory to your agents with the advantage that you can share memory between agents. However, the same latency and other design constraints apply as when addressing any other kind of distributed data architecture. Your mileage may vary, so decide what kind of problems you’re trying to solve.
Is Memory Just a Model Problem?
Maybe. Some have argued that agent memory is fundamentally a model limitation that will go away as soon as models become sufficiently advanced. In this view, assuming a near infinite context window, a sufficiently advanced model could sift through irrelevant or conflicting information, identify what matters, and produce the correct behavior. Extending this idea further, models might even be fine-tuned on the fly as new information becomes available.
This is an active area of research, but there are significant challenges. As discussed earlier, increasing context size dilutes attention. What we observe, is that as more information is added, models become less reliable at identifying what is actually important. While it’s theoretically possible that future models will handle this a lot better, today’s systems still struggle and critical details often get lost in large context windows.
Online fine-tuning presents its own issues. It is computationally expensive, slow, and difficult to do reliably in real-time systems. While techniques like distillation can move retrieved knowledge into model weights, in practice, the memory patterns we’ve discussed here are typically faster, more cost-effective, and more reliable in current systems.
Failure Modes
As you’ve no doubt encountered, agent memory introduces a broad set of failure modes. Most of them aren’t about not having enough memory, but rather too much of the wrong memory. Sometimes the agent simply doesn’t retrieve the right thing, even though it’s stored (retrieval miss). Other times it pulls in loosely related or completely irrelevant context (retrieval noise), which makes reasoning worse instead of better. As memory grows, you also run into context overload, where too much information dilutes attention and important details get lost.
Things get even messier when memories conflict or become stale over time, since models aren’t great at resolving contradictions or knowing what to ignore.
Even when the right memory is retrieved, the model can still misinterpret or misapply it, which turns a retrieval problem into a reasoning problem. On top of that, memory systems tend to accumulate junk like bad outputs, hallucinations, or low-quality data, which degrades performance over time.
In practice, the challenge isn’t just storing information, it’s deciding what matters, keeping it clean, and using it correctly. Whatever your agent memory implementation, you need to figure out a way to review it, clean it up, and analyze it for correctness. Maybe you have a memory review agent or maybe you’re just going through markdown files. Whatever it is, if you are not cleaning up your agent’s memory, it’s going to start to get bogged down and making big mistakes.
Takeaways
I think the most important thing to understand about agent memory is that it is not settled. it’s an active area of research and there are not established best practices for doing it well.
This means there’s an opportunity for you to experiment and break ground. You might think, “Oh, I don’t have the bandwidth to deploy an autonomous agent and databases as a service and a scaled vector DB.” but you can experiment with these patterns right at your desktop. CLI agents are perfect for this type of work. you can easily create agent instruction files, skills, and all the kinds of memory documents we described above and see how they work for your use cases. As you record memory and markdown files, you can review them for correctness and develop tools for automated review. An outstanding way to really internalize how agent memory works is to set up an OpenClaw instance. There are risks so make sure you’re following rigorous security practices and probably don’t do this on your work machine/network unless you have approval.
OpenClaw is an open-source agent framework that’s particularly useful for understanding how agent memory works in practice. It turns a language model into a persistent system with access to tools, files, and external services. You end up dealing with a lot of real memory problems like what to store, how to retrieve it, and how to keep it consistent.
You can experiment with different memory patterns and see the tradeoffs and failure modes of agent memory systems play out in real workflows. Also, it’s really fun. I named my instance Skippy after the AI in the Expeditionary Force novels and he’s a huge jerk.
So experiment, develop use cases, build stuff, and test new patterns for agent memory. Now is a rare opportunity to push the frontier of a technology that’s going to change our industry.