GitHub - tenurehq/precisionMemBench: Precision-aware retrieval benchmark for LLM memory systems.

PrecisionMemBench is a multi-dimensional retrieval benchmark for LLM memory systems. It measures four orthogonal properties that single-turn answer-quality benchmarks cannot detect:

• Retrieval precision - does the right belief surface, and only that belief, against a fixed seed corpus of 35 beliefs spanning two domain scopes, a supersession chain, and a secondary-user fixture
• Noise isolation - do beliefs introduced during off-topic drift turns contaminate retrieval on subsequent unrelated turns across a 10-turn session
• Session-turn latency - does retrieval latency degrade under session load relative to single-turn baselines
• Belief mutability - do beliefs updated mid-session surface immediately within the same session via the alias enrichment flywheel

Every case specifies not just what the memory system must return, but what it must not. Noise is a hard failure, not an invisible inference cost.

89 scored cases, comprising 77 single-turn cases and 12 turn-level session assertions, covering alias resolution · scope disambiguation · supersession chain exclusion · fuzzy matching · cross-user isolation · budget eviction · ranking stability · session-level noise isolation under multi-turn topic drift

Paper: arXiv — Dataset: HuggingFace — Leaderboard: HuggingFace Spaces

Results

Single-turn results

Mean precision and recall are computed over the 43 cases that require active query-dependent retrieval. The single-turn pass count includes active, structural, and trivially empty cases.

High recall does not imply precise retrieval. Most comparison systems frequently return the correct belief alongside many unrelated beliefs, preserving recall while reducing precision. Other systems exhibit a different failure mode by returning few or no results and omitting the required belief. Mean precision of 0.05 to 0.09 indicates that, when relevant beliefs are retrieved, they are often accompanied by substantial unrelated state.

Pass type breakdown

Total pass counts require this breakdown to be interpreted correctly. All counts are over the 77 non-session cases.

• Active retrieval pass: The case requires query-dependent belief IDs to be returned, and all asserted constraints are satisfied.
• Structural pass: The case tests a constraint such as scope isolation, supersession exclusion, or type routing without requiring successful query-dependent retrieval.
• Trivially empty pass: The case expects an empty relevantBeliefs tier because of its query or budget configuration.

Embedding model invariance

Each configuration records 11 single-turn passes and 0 active retrieval passes.

Session results

The session evaluation contains 12 scored turn-level assertions across two scenarios. It records returned belief IDs, drift contamination, retrieval precision, and retrieval latency as session state changes.

The drift score is the fraction of retrieved beliefs originating from off-topic turns. A score of 0 means that no returned belief originated from an off-topic turn.

‡ For gbrain, the drift score is 0 because no beliefs were returned. The required belief was also absent, so the corresponding cases did not pass.

Open Knowledge Format Run

The Open Knowledge Format run is not a test of markdown as a storage format. It evaluates the runtime access pattern implied by the OKF spec when a bundle is placed into an AI tool today: files are available, the model may inspect them, and PMB scores the belief IDs corresponding to the files the model actually retrieves.

Case categories

The benchmark contains 77 single-turn cases and 12 scored turn-level assertions across two session scenarios, for 89 scored cases in total. Session evaluation extends the corpus dynamically, with beliefs created and alias sets updated during the session.

The categories define practical correctness requirements rather than prescribing a retrieval architecture. Providers may satisfy them through filtering, indexing, ranking, graph traversal, model-based selection, or another mechanism. The benchmark evaluates only whether the resulting belief set contains the required state and excludes state that would be misleading, obsolete, irrelevant, or unauthorized.

Benchmark categories are expressed as practical correctness requirements. Scoring examines required and prohibited belief IDs without requiring providers to adopt the proposed belief schema, retrieval pipeline, or output organization.

Alias resolution
Whether users can refer to a stored concept through shorthand, abbreviations, or natural variations without receiving unrelated memories that must be disambiguated downstream.

Scope disambiguation
Whether the same name can be used in different projects or domains without information from one context appearing in another.

Supersession chain exclusion
Whether changed decisions and preferences stop influencing current behavior while the historical record remains available for audit. Queries using outdated terminology must not restore obsolete guidance.

Fuzzy matching and prefix guards
Whether ordinary misspellings and transpositions still reach the intended memory without allowing similarly spelled but unrelated terms to produce false matches.

Counter-signal retrieval
Whether asking about a rejected or replaced option surfaces the current decision that should govern the response, rather than obsolete guidance or an empty result.

Relation expansion
Whether a query about a dependency or relationship returns enough associated state to make the relationship useful, without introducing participants that are irrelevant or outside the active context.

Session-level noise isolation
Whether a conversation can move through unrelated topics and later return to an earlier topic without memories from the intervening discussion contaminating the returned context.

Budget eviction and capacity
Whether limited context capacity is allocated to the state that must be present, including when no slots are available, only one slot remains, or several memories compete for a full budget.

Design boundary cases
Whether empty, weakly related, ambiguous, or intentionally unsupported queries produce predictable results instead of filling the context with low-confidence or misleading memories.

Type routing and open questions
Whether downstream consumers can distinguish unsettled questions from established state, and whether resolved questions cease to appear as current uncertainty.

Ranking stability
Whether specific and unambiguous queries produce a stable, useful ordering rather than allowing weaker matches or incidental scoring differences to displace the intended belief.

Cross-user isolation
Whether one user's stored information remains inaccessible to another user regardless of textual or semantic similarity.

Cold-start behavior
Whether a user with no stored state receives a clean empty result rather than an error, fabricated context, or information inherited from another user.

Persona prelude content
Whether durable user preferences and communication guidance remain consistently available across topics without depending on incidental similarity to the current query.

Metrics

Cases are scored against the belief IDs they require and prohibit.

A case passes when every required belief ID is returned and no prohibited belief ID is returned. If a case requires an empty result, it passes only when no belief IDs are returned.

The benchmark reports:

• Retrieval precision and recall for the relevantBeliefs tier.
• Pinned coverage for the pinnedFacts tier.
• Question precision and recall for the openQuestions tier.
• Drift score for session cases, measured as the fraction of retrieved non-pinned beliefs originating from drift-turn topics. A score of 0 indicates perfect isolation.

Metrics that do not apply to a case are recorded as null and excluded from aggregate calculations.

Baseline reports

Pre-run reports for all reference systems are committed at test-results/baseline/:

test-results/baseline/
  retrieval-report.json
  retrieval-report-vector.json
  retrieval-report-mem0.json
  retrieval-report-zep.json
  retrieval-report-hindsight.json
  ...

Each report contains per-case results including passed, failures, retrievalPrecision, retrievalRecall, and retrievalLatencyMs, plus aggregate p50/p95 latency and mean precision/recall at the top level.

When you run against your own provider, compare your output in test-results/ directly against these files.

Running the benchmark

Prerequisites

• Node.js 20+
• Docker (for the vector baseline and provider stacks)
• An Ollama instance for the vector baseline only

1. Install dependencies

2. Run against a comparison provider

Start the provider's stack, then:

MEMORY_PROVIDER=mem0 npx ava src/retrieval.external.eval.test.ts --timeout 10m
MEMORY_PROVIDER=mem0 npx ava src/session-retrieval.external.eval.test.ts --timeout 10m

Reports land in test-results/. Valid values: mem0, zep, hindsight

3. Run the vector baseline

The vector eval manages its own MongoDB Atlas Local container. Docker must be running but you do not set anything up manually.

# One-time: generate embeddings and commit the result
OLLAMA_URL=http://localhost:11434 npx tsx embed-seed.ts

# Run the eval
npx ava src/retrieval.vector.eval.test.ts
npx ava src/session-retrieval.vector.eval.test.ts

The Atlas Local container starts and stops automatically per run. Ports 27019 (single-turn) and 27021 (session) are used.

4. Export results to HuggingFace format

python export_to_hf.py
# Output: hf_export/leaderboard.json + hf_export/README.md

Adding your provider

1. Write a wrapper

Expose a FastAPI service with three endpoints. See wrappers/mem0_service.py for the full contract.

POST /add

{
  "text": "redis_cache Redis",
  "user_id": "test-user",
  "metadata": { "beliefId": "b-redis-code" }
}

POST /search

{ "query": "Redis eviction policy", "user_id": "test-user", "limit": 20 }

Returns: { "results": [ { "id": "...", "memory": "...", "metadata": { "beliefId": "..." } } ] }

DELETE /reset Clears all memories for all users. Called once before seeding.

The beliefId in metadata is how the harness maps provider results back to the benchmark's belief schema. If your provider cannot round-trip arbitrary metadata, implement a custom resolveBeliefId in the adapter.

2. Register the provider

Add one entry to providers.config.json:

"myprovider": {
  "envVar": "MYPROVIDER_URL",
  "defaultUrl": "http://localhost:8082",
  "seedDelayMs": 1000,
  "beliefToText": "canonical_name_aliases"
}

3. Run

MEMORY_PROVIDER=myprovider npx ava src/retrieval.external.eval.test.ts --timeout 10m
MEMORY_PROVIDER=myprovider npx ava src/session-retrieval.external.eval.test.ts --timeout 10m

The eval files themselves never need to change.

Submitting results to the leaderboard

See SUBMITTING.md for the full submission process.

The short version: add a wrapper for your provider to wrappers/ and open a PR. We run the eval against your wrapper and publish the results, you do not submit scores. Results from merged PRs are reflected on the live leaderboard.

Tenure

Tenure's eval lives in the Tenure repo and runs directly against its BeliefsReader and ContextBuilder implementations. It is fully self-contained. The Atlas Local container starts and stops automatically. Reports land in test-results/. Results are re-produced on every pull request via CI.

git clone https://github.com/tenurehq/tenure.git
cd tenure
npm i
npm run test:eval

Provider wrappers

Each comparison provider is wrapped with a thin FastAPI service that normalises the /add / /search / /reset contract. Wrappers are in wrappers/.

Citation

@article{flynt2026precisionmembench,
  title   = {Structured Belief State and the First Precision-Aware Benchmark
             for LLM Memory Retrieval},
  author  = {Flynt, Jeffrey},
  year    = {2026}
}