GitHub - corbym/locomo-recordari: LoCoMo benchmark harness for Recordari

LoCoMo benchmark harness for Recordari — evaluates Recordari's knowledge graph API as a memory backend on the LoCoMo long-conversation benchmark.

Built on top of the microsoft/Memora research framework (MIT licence).

Results

This repo runs the configuration the Memora paper specifies. The paper (arXiv:2602.03315) splits two roles, and the config here matches both:

Memora's own configs ship exactly that split — see this repo's initial commit (dec3f8f), which vendors Memora unmodified, where eval.model is gpt-4o-mini commented "same as Mem0 and Nemori paper". Paper and code agree, so there is no ambiguity about which judge produced the paper's Table 1.

The Recordari rows below add use_combined_user: true, top_k: 30 with 1-hop edge expansion — retrieval settings on the Recordari side, which the paper does not govern — and exclude category 5 (adversarial), as Memora and most published LoCoMo results do.

Judge model — read the gpt-4o-mini column. That is the paper's judge and the repo's default. The gpt-4.1-mini column is kept because earlier published Recordari figures used that judge; both sets of files ship here so the two are directly checkable against each other.

On identical response text, gpt-4o-mini is the harsher of the two by 0.0097 on Phase 1 and 0.0169 on Phase 2.

Repeat passes of the same judge over the same answers are not perfectly stable, despite temperature=0.0, seed=42 (metrics/llm_judge.py): the two gpt-4o-mini passes over Phase 1 gave 0.8071 and 0.8065 (0.0006 apart, ~1 question), and the two gpt-4.1-mini passes over Phase 2 gave 0.8305 and 0.8344 (0.0039 apart, ~6 questions). Treat differences below ~0.004 as noise.

Every Recordari figure links to the score file it comes from. All LLM scores use the gpt-4o-mini judge and gpt-4.1-mini answering model unless otherwise noted.

Four-arm ablation (Recordari extraction architecture × answer-context format):

Notes on the arms:

• Phase 5 header-only is a labelled negative result. Date-header grouping without source text costs ~−0.025 overall. It is published as an arm, not discarded, because it isolates that the grouping structure alone does nothing: the source text is what moves the needle on cat4.

• Phase 5b is a research configuration, not the product configuration. It locates the extraction loss: adding verbatim source segments to the answer context recovers +0.043 on cat4 (the extraction-discard penalty the facts-only arm carries), but at roughly double Memora's answering context (~70k vs ~36k chars/question). The chars/q column is in the table because the efficiency gap belongs in the comparison, not in a footnote.

• The residual gap to Memora S decomposes by category (Phase 5b vs in-harness Memora S, −0.014 overall): cat1 (multi-hop) accounts for 57% of it (−0.042), cat2 (temporal) 29% (−0.019), cat4 (single-hop) 19% (−0.005). Cat3 offsets +0.007 in Recordari's favour.

• Cat3 (open-domain) ordering is monotone across all four arms: facts-only 0.6250 → header-only 0.6042 → verbatim 0.5625 → Memora 0.5521. More context in the answer reliably hurts inferential/hypothetical questions. This is the most directly relevant finding to what Recordari is built for — extracted facts outperform raw source text on exactly the question type that most resembles real knowledge work.

Historical comparison including BLEU/F1 and earlier phases:

All Recordari rows: 1,540 scored questions across 10 conversations. BLEU and F1 are judge-independent — computed from the answer text — so each phase has a single value across both columns.

One difference the model table above does not capture: Memora's §5.1 backbone also drives its prompt-based policy retriever. Recordari has no equivalent — the adapter retrieves through the Recordari API (search + recall, top_k: 30, 1-hop edge expansion). llm.model governs the LLM calls the adapter does make: extraction and answering.

Recordari is below both published Memora system configurations under either judge. Phase 1 stores raw conversation batches; Phase 2 uses an LLM to extract discrete labelled nodes per session, resolving relative dates to absolute at write time. BLEU/F1 are lower for Phase 2 because those token-overlap metrics reward verbatim storage; paraphrasing into clean facts necessarily loses surface overlap, so the LLM judge is the meaningful comparison.

Phase 2's aggregate gain over Phase 1 shrinks under the paper's judge, from +0.0182 (gpt-4.1-mini) to +0.0110 (gpt-4o-mini). At about 17 questions out of 1,540 that is above the ~0.004 noise floor, but it is a modest effect and it is not what makes Phase 2 interesting. The per-category breakdown below is where the real difference lives.

Correction (2026-08-13). Earlier versions of this README compared Recordari's 0.831 against "Memora 0.801" and claimed a win. That was wrong. 0.801 is a component-ablation row from Table 3 of arXiv:2602.03315 ("primary abstraction, with update"), not a published system result. Memora's system scores are in Table 1: 0.849 (S) and 0.863 (P). Recordari does not beat either under any judge.

Correction (2026-08-17 — parity claim). Against the in-harness Memora S run (same harness, same gpt-4.1-mini answering model, same gpt-4o-mini judge), Recordari v5 trails on multi-hop (−0.042), temporal (−0.019), and single-hop (−0.005 in the Phase 5b configuration). Open-domain is the one category where Recordari's extracted-fact representation leads: +0.073 over Memora S in the facts-only arm, with the gap widening further when source text is added to context. The in-harness figures take precedence over the paper comparison for any per-category claim.

Per-category LLM scores. LoCoMo's categories are reasoning types, in the same order the Memora paper's Table 1 uses them. The first two score columns are the paper's judge:

(Category 5, adversarial, is excluded. LoCoMo's five reasoning types are defined in the LoCoMo paper; the labels above were verified against the dataset — category 2 is 77% "When did…" questions with 1.17 evidence turns on average, category 1 averages 3.13 evidence turns with 98% multi-evidence, category 4 averages 1.07 with 95% single-evidence.)

The substantive result is temporal reasoning (cat 2), and unlike the aggregate it barely moves with the judge: +0.106 under gpt-4o-mini (0.757 → 0.863) and +0.112 under gpt-4.1-mini (0.760 → 0.872) — an order of magnitude above the noise floor, and consistent across two independent graders. That is exactly the mechanism Phase 2 changes: extraction resolves relative dates ("last Tuesday") to absolute ones at write time, so the answerer is handed a date instead of having to infer one from surrounding dialogue. Multi-hop also improves (+0.043 / +0.064) — discrete labelled facts are easier to chain — though by less. Both are controlled before/after comparisons within one harness, independent of any external comparison, and they are the findings this repo actually supports.

Single-hop (cat 4) went the other way — −0.033 (gpt-4o-mini), −0.032 (gpt-4.1-mini). These are direct detail-recall questions answerable from one turn, and extraction summarises detail away: a verbatim batch keeps "salads, sandwiches and homemade desserts" where an extracted fact records "dinner". Because cat 4 is 841 of the 1,540 questions and cat 2 only 321, that regression cancels most of the temporal gain once everything is averaged into a single number. The aggregate is therefore the least informative view of this comparison, whichever judge computes it.

Where the gap to Memora actually is

Memora S was reproduced in-harness on 2026-08-17 (same dataset, same gpt-4.1-mini answering model, same gpt-4o-mini judge, category 5 excluded — 1,540 scored questions). In-harness figures take precedence for any cross-system comparison; the paper column is kept for reference.

Config used for the Memora S run (parameters.yaml): retrieval.strategy: semantic, use_segments_as_episodic: true (the segment-as-episodic variant, not LLM-summarised episodes), use_combined_user: true. The paper's §5.1 does not specify these settings, so the paper-column comparison carries a configuration caveat. Recordari v5 also runs use_combined_user: true, so the head-to-head between the two systems is on equal footing within this harness.

For v5 (facts-only), weighting by question share, the −0.034 overall gap decomposes as: cat4 −0.026 (76%), cat1 −0.008 (23%), cat2 −0.005 (15%), cat3 +0.005 (−13%). After Phase 5b closes cat4 to within noise (−0.005), the residual −0.014 gap decomposes as: cat1 57% (−0.042), cat2 29% (−0.019), cat4 19% (−0.005), cat3 offsets +0.007. Cat1 (multi-hop, synthesis) is the dominant remaining gap, and adding more context did not move it: Phase 5b doubled the answer context versus v5 with zero effect on cat1. The residual is a synthesis problem, not a retrieval-coverage problem.

An earlier version of this section compared against the paper's quoted figures and concluded "the entire shortfall is single-hop" and "Recordari is marginally ahead on multi-hop and temporal". Against measured in-harness figures both of those claims are false: Recordari trails on all four categories except open-domain. The earlier conclusion was wrong in direction for cat1 and cat2, not merely in magnitude.

Earlier attempt

An earlier Phase 2 attempt (extraction prompt v1) scored 0.768 with a gpt-4.1-mini judge — a regression against the 0.8344 in the second LLM column above, driven by temporal collapsing to 0.620. It was traced to two bugs: the answer formatter dropped occurred_at before it reached the answerer, and cross-run graph contamination leaked stale nodes into answers. Both were fixed before the run published here, and fixing the first is what took temporal from 0.620 to 0.872 — consistent with the mechanism, since a dropped date is precisely what breaks a temporal answer. That attempt's output is not published in results/, which holds only the two runs behind the table above.

Phase 3 — v3 hybrid extraction (instructive)

v3 changed the extraction schema: description became a verbatim quote from the source text (to preserve concrete detail for single-hop retrieval), with why_matters carrying the interpretive gloss and serving as the primary semantic search surface. The hypothesis: verbatim storage would recover Phase 2's single-hop regression while keeping temporal gains, because the exact phrasing from the conversation is what single-hop questions ask about.

cat4 improved as predicted (+0.027). But cat2 collapsed (−0.178): verbatim description preserves relative dates as stored text ("last Tuesday", "a few weeks ago"), undoing exactly the date-resolution mechanism that drove Phase 2's +0.106 temporal gain. Because single-hop is 841 of the 1,540 questions and temporal only 321, the cat2 collapse dominates and the overall score falls.

What this teaches: the description field is both a retrieval surface and the text the answerer reads. Verbatim storage has two effects simultaneously: it improves literal-detail recall for single-hop questions, and it stores unresolvable relative time references for temporal ones. The two are not separable by changing why_matters alone. The natural direction: keep description verbatim for non-temporal facts and add a deterministic post-processing step to resolve relative expressions before storing. That is what v5 implements — see Phase 4 below.

The Phase 3 run is not published to results/ as a linked score pair — it is a development artefact, not a finished run. The score file is at app/locomo/results/recordari_20260814_182947/recordari_scores.json and is committed here for auditability.

Phase 4 — v5 (verbatim + normaliser)

v5 separates what prompts cannot hold simultaneously. Phase 2's LLM extraction resolved relative dates at write time (+0.106 on temporal) but paraphrased detail away (−0.033 on single-hop). Phase 3's verbatim storage recovered that detail (+0.027) but stored relative dates as written (−0.178 on temporal). v5 keeps Phase 3's verbatim description and adds a deterministic harness-side step: surviving relative expressions ("last Tuesday", "a few weeks ago") are replaced with natural absolute phrases derived from occurred_at before calling remember().

The result is a composition that was previously out of reach: cat4 (single-hop) beats Phase 2 at the same time cat2 (temporal) recovers near it — +0.023 and +0.146 respectively over what each category showed in Phase 3. The overall gain (+0.005 to 0.8221) is at the ~0.004 noise floor and is not the finding; the finding is that the Phase 2 / Phase 3 trade-off is no longer forced.

v5 bundles two independent changes. A gate ablation on a single conversation (conversation 0, 37 cat2 questions) measured each one's contribution to temporal reasoning before the full run:

The [event: …] prefix in the answer context contributes +0.054 on cat2; the deterministic normaliser adds a further +0.189. These come from single-conversation gate runs, not a purpose-built ablation arm — treat them as directional.

The residual −0.031 gap to Phase 2 on cat2 reflects Phase 2's LLM paraphrasing descriptions to exact absolute dates, which the deterministic normaliser cannot replicate when occurred_at is only month-precise. cat4 holds at 0.8537 (+0.023 vs P2): verbatim descriptions preserve the concrete detail the normaliser does not touch.

Temporal stats: 2,579 nodes, 19.5% pre-normalisation unresolved expression rate, 503 normalised (all affected nodes), 0.7% no-occurred_at floor. Gate verified post-normalisation rate ≤ floor before the full run.

Run: v5_full, 2026-08-16, Recordari 1.5.0, prompt v5, skill recordari-skill-f3402c15.md. Scores at app/locomo/results/recordari_20260816_154430/recordari_scores.json.

Comparability

The judge matches. Memora's published numbers were judged by gpt-4o-mini (paper Appendix B, and eval.model in their configs), and so are the Recordari figures in the first LLM column. The judge is not a confound here.

Worth stating plainly why that matters: the judge shifts Recordari's score by 0.0097–0.0169, about the same size as the +0.0110–0.0182 Phase 1 → Phase 2 difference this repo exists to measure. A LoCoMo number quoted without naming its judge omits an effect as large as the result.

Memora S has been reproduced in-harness (2026-08-17, run memora_semantic_20260817_092603, scores at app/locomo/results/memora_semantic_20260817_092603/memora_semantic_scores.json). Same dataset, same judge, same answering model. In-harness figures take precedence over paper-quoted ones for any cross-system claim; both columns are kept in the gap table above. Memora P has not been run in this harness.

Category 5 (adversarial) is excluded, per the convention Memora and most other published LoCoMo results follow.

Recordari version, and why these numbers may drift

These runs measured a hosted service, not a pinned artefact.

Both phases ran on the same server build, and no deploy landed during either run. Four deploys landed between them, on the evening of the 24th; none changed the tool surface, so the Phase 1 → Phase 2 comparison is not confounded by an API change. (The adapter did not record this at the time — it was reconstructed afterwards from the deploy history. Future runs should capture it directly.)

ServerVersion tracks the tool surface, not behaviour. It is bumped when tools are renamed, parameters change, or response shapes change — deliberately, so agents know to refresh a cached tool list. Retrieval quality can move without it changing. A concrete case: a change making occurred_at part of the embedding text and surfacing it in read results deployed on 2026-07-25, roughly four hours after Phase 2 finished, under the same 1.4.2. Date visibility in surfaced results went from ~0% to 97%, though semantic recall on date-phrased queries only reached 0.41.

What has actually changed since, and what it means for a re-run. Auditing every commit to the server between these runs and 1.5.0, three changes touch anything the benchmark exercises:

• occurred_at in the embedding (above). The one worth flagging: it alters what vector search ranks, so retrieval at top_k: 30 can differ, and it lands on the temporal category that carries the headline result. A fresh run would also embed newly-written nodes with the date included rather than depending on the one-off backfill. Effect size and direction are untested — dates in the embedding should help date-phrased queries, but the probe above only reached 0.41 on exactly those, and the harness already surfaces occurred_at itself, which makes part of the change redundant here.
• Lifecycle state on lean/digest result lines — adds a field to retrieved context. Marginal.
• Tool-surface consolidation to 1.5.0 — 20 tools folded into 16, with the retired ones redirected rather than removed. No functional change, and the harness calls only survivors (search, recall, remember, connect, audit, domains, forget), so a clone still runs today.

Nothing else in that window touches ranking, embedding or retrieval.

So: a re-run today would most likely land close to these numbers, and the consolidation is not a reason to expect otherwise. The single open variable is the occurred_at embedding change, and it is concentrated in one category. Treat these figures as measurements of Recordari 1.4.2 as it behaved in late July 2026 — a different result on a re-run is expected behaviour when benchmarking a live service, not automatically a discrepancy.

Published run data

The three full runs behind the table ship in this repo, so you can audit or re-score them without paying to regenerate the answers.

Phase 1 — raw batches · app/locomo/results/recordari_20260724_123147/

Phase 2 — skill-faithful extraction · app/locomo/results/recordari_20260725_091956/

v5 — verbatim + normaliser · app/locomo/results/recordari_20260816_154430/

Note the unsuffixed recordari_eval.json / recordari_scores.json are each run's inline eval and do not use the same judge in both directories. Phase 1's used the then-default gpt-4o-mini; Phase 2's was overridden to gpt-4.1-mini on the command line. Neither file records which model judged it, so that attribution comes from the config history — eval.model was gpt-4o-mini from the initial commit until 0dd3fb6 (2026-08-13) — and from the override documented in that commit. It is consistent with the scores themselves (Phase 1's 0.8071 sits 0.0006 from the known gpt-4o-mini pass; Phase 2's 0.8305 sits 0.0039 from the known gpt-4.1-mini pass), but it is inference, not metadata. The suffixed files are unambiguous — prefer those. They are what every number in this README links to.

What each file contains:

• recordari_output.json.gz — every question, the gold answer, the generated answer, and the memories retrieved for it. 1,986 rows (1,540 after excluding category 5) across 10 conversations. Gzipped because the raw files are 436 MB and 550 MB, over GitHub's 100 MB per-file limit.
• recordari_eval_<judge>.json — the per-question verdicts behind a score file.
• recordari_scores_<judge>.json — overall and per-category aggregates.
• run_id.txt — the run id the Recordari benchmark domains were named from (bm_<user>_<run_id>, see providers/recordari/add.py). Phase 1 is 20260724_103432, Phase 2 phase2_v2_full. That workspace has since been torn down, so these are provenance only — they will not resolve against the API.

Repaired before publication (2026-08-14): 8 Phase 2 answers. While re-judging, 8 of the 1,540 Phase 2 rows were found holding the literal string ERROR: LLM call failed. instead of an answer. Cause: an OpenAI outage mid-run failed 8 answer-generation calls, all in category 4 of conversation 2. They were re-answered once the outage cleared, and those recovered answers are what the run's eval — and the published 0.831 — actually scored; only the saved recordari_output.json kept the pre-recovery placeholders. Left alone, the shipped answers would have disagreed with the shipped scores and re-scoring would have landed ~0.005 low.

The recovered answers survived in recordari_eval.json, so they were restored into the output file with restore_failed_answers.py — which touches only the response field of placeholder rows, matched on (conversation, question, category, gold answer), and refuses to write if any placeholder is unmatched or survives. All 8 matched.

The repaired file is the one published here, and it verifies: re-scoring it reproduces BLEU 0.3700 and F1 0.4396, identical to the originally published values. Those are computed from the answer text with no LLM involved, so an exact match is proof the restored responses are the ones originally scored. Re-judging it with gpt-4.1-mini gives 0.8344 against the 0.8305 published at the time — a 0.0039 spread that is judge run-to-run noise, not a data difference. Phase 1's file never had this problem and reproduced on the first attempt.

Re-score them yourself — eval_only.py reads the .gz directly and defaults to the paper's judge:

cd app/locomo

# paper's judge (gpt-4o-mini) — reproduces 0.8175
python eval_only.py --output-dir results/recordari_20260725_091956

# any other judge
python eval_only.py --output-dir results/recordari_20260725_091956 --judge gpt-4.1-mini

That pays for judge calls only. The expensive part — ingesting 10 conversations into Recordari and generating answers to all 1,986 questions, 1,540 of which are scored — is already done, and preserving it is the point of publishing these files.

What's closed

The LoCoMo experimental arc is complete. These threads are explicitly closed — do not retry.

Cat4 single-hop. Done. Phase 5b reaches 0.8966; the remaining gap to Memora S (0.005) is within the judge noise floor (~0.004). The extraction-discard penalty is fully quantified and has been recovered at ~2× answer-context cost.

Date resolution in the extraction prompt. Closed. Three prompt variants (v3 appended, v4 inline, v5 simplified) all produced ~25% raw unresolved relative expressions — verbatim instruction and replace-in-place instruction are logically incompatible, and the model won't reliably do both. The deterministic normaliser in temporal.py is the fix, and it is shipped.

Cat1 via more context. Measured zero effect. Context grew ~31% entry-count and ~13% character-count from v5 to Phase 5b. Cat1 score was unchanged at 0.7837 in both arms. Cat1 is a synthesis problem, not a retrieval-coverage problem.

Query-intent classifier (per-question-type top_k). Deprioritised. The measurement establishes cat1 is synthesis-limited, not retrieval-limited. A classifier would not address the root cause. Reopen only if a new synthesis approach is attempted.

Open threads that justify new benchmark spend: the cat3 ordering finding (why extracted facts outperform raw source text on inferential questions — relevant to product positioning), and LongMemEval (a new benchmark, not a continuation of LoCoMo).

Setup

Prerequisites: Python >= 3.10, a Recordari workspace key, an OpenAI API key.

git clone https://github.com/corbym/locomo-recordari
cd locomo-recordari
pip install -r requirements.txt
pip install python-dotenv httpx jinja2

There is no pyproject.toml/setup.py, so the vendored memora library is not installed as a package — put it on the path instead:

export PYTHONPATH=$PWD/src        # Windows PowerShell: $env:PYTHONPATH = "$PWD\src"

Create a .env file at the repo root:

RECORDARI_API_KEY=your_recordari_workspace_key
OPENAI_API_KEY=your_openai_key

Download the LoCoMo dataset and place it at app/locomo/data/locomo10.json:

# https://github.com/snap-research/locomo/tree/main/data
mkdir -p app/locomo/data
# place locomo10.json here

Running

cd app/locomo

# Phase 1 — raw conversation batches (baseline)
RECORDARI_RUN_ID=phase1 python run_recordari.py openai.api_type=openai

# Phase 2 — skill-faithful LLM extraction
RECORDARI_RUN_ID=phase2 python run_recordari.py \
    openai.api_type=openai \
    extraction.enabled=true \
    extraction.model=gpt-4.1-mini

# Skip ADD, reuse an existing run's memories
RECORDARI_SKIP_ADD=1 RECORDARI_RUN_ID=phase1 python run_recordari.py openai.api_type=openai

# Re-evaluate with a different judge model
python eval_only.py --output-dir results/recordari_YYYYMMDD_HHMMSS --judge gpt-4.1-mini

# Clean up stale benchmark domains
python cleanup_stale_bm_domains.py --dry-run
python cleanup_stale_bm_domains.py
python cleanup_stale_bm_domains.py --csv domains.csv --purge --force

For full configuration options see app/locomo/README.md.

Repo structure

locomo-recordari/
├── app/locomo/
│   ├── providers/recordari/   # Recordari adapter
│   │   ├── client.py          # MCP-over-HTTP client (remember, search, forget, audit, connect)
│   │   ├── add.py             # ADD phase: Phase 1 batch + Phase 2 extraction
│   │   ├── search.py          # SEARCH phase
│   │   ├── extractor.py       # LLM extraction orchestrator
│   │   └── extraction_prompt.py  # Versioned extraction prompt (PROMPT_VERSION)
│   ├── run_recordari.py       # Main entry point
│   ├── eval_only.py           # Re-run evaluation on existing output
│   ├── cleanup_stale_bm_domains.py  # Purge stale benchmark domains
│   └── conf/config.yaml       # Hydra config (extraction, eval, openai blocks)
└── src/memora/                # Upstream Memora library (unchanged)

Methodology

Phase 1 (raw batches)

Each conversation session is chunked into 4-turn batches and stored verbatim as finding nodes. No LLM is used during ADD. This is a deliberate lower-bound baseline. It is not how Memora ingests: Memora's pipeline is LLM-curated (segmentation, episodic and factual extraction, cue-index generation — the "memory curation" role in the paper's §5.1). The only thing Phase 1 has in common with it is keeping the stored value verbatim rather than compressed.

Phase 2 (skill-faithful extraction)

Each session is passed to an LLM with a structured extraction prompt (see extraction_prompt.py; the published run used PROMPT_VERSION = "v2"). The LLM identifies discrete durable facts and files each as a separate labelled node with node_kind, tags, why_matters, and occurred_at. It also proposes connections between related nodes. This matches how a real Recordari agent is instructed to file memories.

Attribution

• Benchmark framework: microsoft/Memora (MIT)
• Dataset: snap-research/locomo — Maharana et al., 2024
• Memory API: Recordari