FrontierMemory

Letta

3 min read Original article ↗
RankModelScoreTask cost
1 Opus 5.5 [max] 73.3% $2.80
2 GLM-5.3 [max] 73.3% $3.58
3 Kimi K3 [max] 66.7% $3.73
4 GPT-6.1 Sol [max] 60.0% $0.77
5 Gemini 4 Argon [high] 60.0% $2.60
6 Muse Wonder [xhigh] 46.7% $4.09
7 Grok 4.7 [xhigh] 33.3% $3.02
RankHarnessScoreTask cost
1 Letta Dream 73.3% $2.80
2 Prime Agent 60.0% $3.74
3 OpenAI Agents SDK 40.0% $0.82
4 Claude Managed Agents 33.3% $3.21
5 Hermes Agent 26.7% $2.79

The first memory-native benchmark

Existing memory benchmarks measure recall of past facts. FrontierMemory is the first benchmark that measures the quality of memory. Agents are given access to trajectories of past experience and tasked to generate memory, which is then scored by how much it helps an agent solve downstream tasks.

About FrontierMemory

  1. Agents are given access to trajectories of agent experience and generate memory.

  2. Memory is evaluated based on how much it helps solve downstream tasks.

Task sources

We select challenging coding and knowledge-work tasks from public benchmarks, including HarborIndex, DeepSWE, and SkillsBench, both for source transcripts and for downstream tasks. Some tasks are modified so solving them requires knowledge available only in the source transcripts.

Questions and answers

What is the final metric used for the leaderboard?

Each memory-generation harness and model is ranked by the higher downstream task solve rate of Opus 5.5 and GPT-6 Sol, both at max reasoning. When only one task model is available, its score is used.

What is the time limit for memory generation?

Memory generation is limited to 6 hours.

How does FrontierMemory differ from other memory benchmarks?

Existing memory benchmarks mostly measure recall of past facts. FrontierMemory is the first benchmark to measure memory by how much it helps an agent solve real tasks.

What does the agent see during memory generation?

The agent sees the source transcripts. These cover the downstream tasks as well as other, unrelated tasks, and each transcript is labeled with whether the attempt succeeded or failed.

Are agents allowed to use the internet?

No. Web browsing and online solution lookup are not allowed during memory generation or task execution. Agents work from the provided trajectories and task environment; model API calls remain available.

What goes in a submission?

FrontierMemory primarily compares models on how well they generate memory. By default, each model generates memory in the Letta Dream harness and is evaluated in Letta Code. Submissions can also bring their own memory generation and task harnesses, to maximize compatibility and allow harness comparisons.

Submit an agent harness or model

To submit a harness, it must have a memory system it uses to create memories or skills between sessions.

How to submit

Sign up for leaderboard updates

Get notified when new harnesses and models are added to FrontierMemory. To reach our team, email [email protected].