Five failure modes around fact retirement in agent memory, with measurements

Background

A memory store for agents has to handle a fact being replaced. In March the project uses Postgres; in June it moves to SQLite. A store that keeps both returns both, and the model has to pick.

Several systems already retire the older fact at write time. They differ in how they decide that a new fact replaces an old one:

This post covers five problems we ran into around that step while running our own memory store on our repositories, and the measurements behind each change.

1. Negations pass similarity thresholds

agentmemory’s tokenizer splits on whitespace and drops tokens of two characters or fewer. With it reimplemented, Jaccard on three negation pairs:

All three exceed 0.7, so the newer statement retires the older one. If the negation arrives second, that is the correct outcome. If the affirmative arrives second (the old behaviour re-asserted after a reversal), the correction is the one that gets retired.

Our rule has the same problem. It compares titles: each fact has a short subject title, and if one title’s tokens are a subset of the other’s, the newer fact retires the older. Negation words are not stop words, so “Push gate blocks default branch” is a subset of “Push gate no longer blocks default branch”. This retired one of our own decisions.

The change: if every token the longer title adds is a polarity token (not, no, never, none, nor, without, cannot, longer, unable, and the stems isn, aren, wasn, doesn, didn, hasn, couldn, wouldn ...), both facts stay active and the pair is listed as a conflict. can is excluded because "Can the gate block the branch" is a normal title. Numbers survive tokenization, so "$99" and "$149" titles are not subsets of each other and needed no separate guard.

Before using titles, we measured embedding similarity for the same decision on a labelled set:

No threshold separates those.

Margin, ratio, margin/sd, z-score and an exponential tail were also tried. All five separated worse than raw cosine.

2. A write can contradict a decision that nobody retrieved

An agent stored a finding recommending removing MCP tool-argument descriptions to save tokens. An active decision in the same store had rejected that, with a stated condition for reopening it. The write went through without any signal.

Retrieval would have found the decision: the query “should we trim MCP tool descriptions to save tokens” returns it at rank 1. The agent never ran that query; it had searched for startup performance. The write is the one point where the new item and the existing decision are both available, so the check went there: each write is scored against active decisions, and a match above threshold is returned in the write result.

The first scorer was raw cosine against a fixed threshold. It measured AUC 0.972 on a labelled set stratified by score band. On a set whose negatives were sampled uniformly from real writes it measured AUC 0.836, with 0% recall at the fire rate we had shipped. The stratified set had over-represented pairs a fixed threshold handles well.

The replacement is CSLS (cross-domain similarity local scaling): subtract each side’s mean similarity to its k nearest neighbours (k=10) before comparing, which stops broad decisions from matching every write. The threshold is the store’s own 90th percentile, not a fixed constant.

It only warns. The gap between a true match and the best false match is about one standard deviation, which is not enough to justify refusing a write. It needs 25 or more active decisions to compute a percentile.

3. When one fact is corrected, its batch is likely also wrong

Facts are often written in batches: a session summary, an extraction pass, an imported document. When one item from a batch is deprecated or rejected, or reported as having caused a correction, the other items from the same write commit, with the same source label or citing the same evidence, are set to needs_review. This lowers their ranking by 6% and marks them in the viewer.

Limits: only active, fresh items are touched; at most 12; no recursion, since a review flag is not a correction.

Related behaviour elsewhere: agentmemory marks graph nodes sharing a source observation as stale when a memory is superseded; cognee cascades on document deletion.

We have not measured how often a flagged sibling is actually wrong.

4. File-change staleness is mostly noise

Facts can cite files and symbols. Kage, memwarden and agents-remember flag or withhold memories when cited code changes; Kage also lists the memories a working-tree diff invalidates (kage pr check).

Our first version flagged a fact whenever a cited file changed. Measured on our own store:

Items are written about files that change often, so a per-edit flag put a large share of the store in review permanently. The check was detection-only for that reason.

The current classification counts only these:

• the cited path no longer exists in the tree
• the cited symbol’s signature hash no longer exists (a rename is suggested only when exactly one candidate is close enough)

It ignores edits, git-reported renames, and churn paths (package.json, lockfiles, README.md, CHANGELOG.md, .github/, dist/, build/, coverage/). Replayed on the same store, 44 items remained, about 5% of the 867; 30 of those came from one refactor's renames, which is why rename detection was added. That is small enough to demote automatically at session start.

DocPrism (arXiv 2511.00215) reports a similar pattern for code/documentation inconsistency checks: flag rate 98% to 14% after excluding the benign category.

5. Another session changed code this session read

With several agent sessions on one repository, session A reads createSession, session B changes it, and A then edits on the old assumption.

Three published results:

STORM lists its own limitation: at file level, “two agents editing different functions in the same file trigger a false-positive rejection”.

Related tools: mcp_agent_mail (file reservations and inboxes), EGC (a session bus with path claims), gitmem (a PreToolUse hook that blocks until recalled items are acknowledged).

What we built follows STORM’s approach, applied through hooks in existing agent CLIs (Claude Code, Codex, and others) instead of an orchestrator. Each session records a read-set of symbol://path#Name locators with the hash seen at read time. If another session changes one, the next edit to that file is refused once, with the diff, and the read is released so the retry succeeds. Because it is per-symbol, reading createSession does not block an edit to destroySession in the same file.

It refuses nothing by default. It is off in the current release (5.23); the next release makes the default “shadow”: verdicts are computed and logged, nothing is refused. Enforcement is gated on ≥95% precision over ≥40 adjudicated shadow findings, which has not been reached.

What is on by default: each MCP tool result includes a short list of items other sessions wrote since this session’s previous call.

Effect of retirement on MemoryAgentBench

MemoryAgentBench (ICLR 2026) FactConsolidation writes a stream of facts, rewrites some, and asks for the current value. Same code with retirement on and off, the benchmark’s own harness and scorer, gpt-4o-mini reader, 100 questions, temperature 0.7, single runs:

An independent rebuild of the adapter scored 94/78 and 90/73. With retirement off at 262K, the correct fact was in the top-10 retrieved context for 100 of 100 questions; 73 were answered correctly.

The scorer uses substring match, so an answer listing every candidate value would score. Our answers have a median length of 2 words and a maximum of 8, and exact match equals substring match in all four cells.

Published numbers on the same 262K single-hop split. Only the Knowl row is our run; the others come from different runners on different days:

Reddy & Challaram read the benchmark’s serial numbers at answer time; on LongMemEval, which has no serials, they report no significant gain.

Limitations

• Multi-hop FactConsolidation: 7. Retrieval is the bound: 14 of 100 questions have the gold fact anywhere in the retrieved set, because the query is single-shot. Reddy & Challaram report 27 and 41.
• Time: valid time is set equal to record time. An as-of query returns earlier content but does not return items that have since been retired. Graphiti stores valid and system time separately.
• Reversal phrases: for reversals written under an unrelated title, a warning fires on phrases such as “abandoned” or “no longer” when the sentence also names an existing fact. It fires on 2.4% of writes in our store; all 25 inspected flags were narrative mentions, not live contradictions. It only warns.
• Provenance (observed / stated by a user / inferred) is unset on 73% of real writes. Blocking retirement by provenance was tested on 101 real replacements: it would have blocked 3, all legitimate corrections.

Open question

The title-subset rule requires the writer to name the subject consistently in the title, and misses replacements whose titles are worded differently. It is deterministic, needs no model, and works on a two-fact store. If you know a cheap deterministic signal for “same subject, new value” that does better, I would like to hear it.