Memory Machines — can LLMs make flashcards that last?

Read the report The full research writeup — evaluation, training, grounding, and arena results

Memory systems make memory a choice—but only if you write practice prompts that effectively reinforce those ideas. That often demands more effort or skill than users can muster—and prompts can’t easily evolve or deepen over time.

Could we make memory as effortless as using a highlighter? We explored whether LLMs could convert casual highlights into useful memory prompts. We found that models can usually identify the intent of highlights, but struggle to generate prompts that will survive long-term review.

The Problem

Unlike a typical textbook practice problem, a memory prompt must cue the same retrieval over time: today, then again in three months, then again in a year. Ambiguous or imprecise prompts produce frustration and forgetting, while stable prompts compound. To study this problem, we constructed a dataset of ~1,500 highlight-anchored prompts across 93 sources and labeled their quality with a four-tiered system.

The Boundary

Across every setup, models reliably reject prompts that are clearly off-target (T0). But they struggle with construction quality—specifically, they can’t reliably distinguish prompts that look plausible but degrade in review (T1) from those that need polish but hold up over time (T2). That’s a problem because T1 prompts are expensive failures. They often seem fine at a glance, only to produce frustration and confusion in later review.

We saw the same pattern across elaborate prompting, rubric-based evaluation, preference learning, fine-tuning, and reinforcement learning: the targeting signal responds to optimization; the construction signal does not.

Grounding

Despite that inconsistent judgment, we found a way to approximately evaluate models. Instead of classifying a prompt in absolute terms, a grounded judge compares it to labeled prompts from the same highlight. This relative approach doesn’t need the model to fully internalize our taste; the standard is fixed in the reference set instead. Under grounding, precision rises from 56% to 78%, while recall falls modestly from 86% to 77%. The judge becomes more conservative, but the ones it approves are far more likely to be usable.

Arena Results

Drawing from highlights in the dataset, the grounded judge rated every model’s output; those ratings aggregate into the Elo rankings below. GPT-5.2 ranks first—and still produces unusable prompts ~36% of the time.

19 models scored by a grounded judge on 200 highlights, with cost-sensitive Elo ratings that penalize plausible-looking failures more heavily than obvious ones.

Read the full report

Evaluation, training, grounding, and arena methodology