FrontierMemory

FrontierMemory

by

Benchmarking continual learning through memory.

Run the benchmark Read the blog

Models v1.1

Which models are the best at memory generation?

RankModelScoreTask cost
1 Opus 5.5 [max] 73.3% $3.42
2 DeepSeek V4.1 Flash [max] 71.1% $3.86
3 Kimi K3 [max] 71.1% $4.16
4 GLM-5.3 [max] 68.9% $4.08
5 Gemini 4 Argon [high] 64.4% $1.05
6 Kimi K3.1 Pro [max] 55.6% $3.61
7 Muse Wonder [xhigh] 51.1% $3.56
8 GPT-6.1 Sol [max] 51.1% $4.00
9 GPT-6.1 Terra [max] 46.7% $3.97
10 Grok 4.7 [xhigh] 37.8% $3.70

Mean ±1 sample SD across 3 independent runs

Harnesses v1.1

Which harnesses are the best at memory generation?

RankHarnessScoreTask cost
1 Letta Dream 73.3% $3.42
2 Hermes Agent 64.4% $3.40
3 Claude Managed Agents 55.6% $3.12
4 Prime Agent 53.3% $4.51
5 OpenAI Agents SDK 26.7% $1.86

Mean ±1 sample SD across 3 independent runs

The first memory-native benchmark

Existing memory benchmarks measure recall of past facts. FrontierMemory is the first benchmark that measures the quality of memory. Agents are given access to trajectories of past experience and tasked to generate memory, which is then scored by how much it helps an agent solve downstream tasks.

About FrontierMemory

  1. Agents are given access to trajectories of agent experience and generate memory.

  2. Memory is evaluated based on how much it helps solve downstream tasks.

Task sources

We select challenging coding and knowledge-work tasks from public benchmarks, including HarborIndex, DeepSWE, and SkillsBench, both for source transcripts and for downstream tasks. Some tasks are modified so solving them requires knowledge available only in the source transcripts.

Sample tasks

See all tasks
DeepSWEAwilix Release Tier (Implicit)

Add async initialization to Awilix, where the version bump is determined by applying a policy learned from earlier tasks to a release tier found in project metadata.

 BigCodeBenchWord2Vec Pipeline

Build a Word2Vec component whose interface, preprocessing, and output contract must be inferred from previous runs.

 SkillsBenchNetwork Intrusion Analysis

Compute DAPT2020 network statistics using lessons from failed analyses about counting, entropy, and threat detection.

Questions and answers

What is the final metric used for the leaderboard?

Each memory-generation harness and model is ranked by the higher downstream task solve rate of Opus 5.5 and GPT-6 Sol, both at max reasoning. When only one task model is available, its score is used.

What is the time limit for memory generation?

Memory generation is limited to 6 hours.

How does FrontierMemory differ from other memory benchmarks?

Existing memory benchmarks mostly measure recall of past facts. FrontierMemory is the first benchmark to measure memory by how much it helps an agent solve real tasks.

What does the agent see during memory generation?

The agent sees the source transcripts. These cover the downstream tasks as well as other, unrelated tasks, and each transcript is labeled with whether the attempt succeeded or failed.

Are agents allowed to use the internet?

No. Web browsing and online solution lookup are not allowed during memory generation or task execution. Agents work from the provided trajectories and task environment; model API calls remain available.

What goes in a submission?

By default, submitted models generate memory with Letta Dream and are evaluated on tasks with Letta Code. You can also submit a memory-generation harness through an adapter PR. The email should explain how Letta can access the model. Letta runs three official evaluations. See how to submit.

Submit an agent harness or model

Models use Letta Dream and Letta Code by default. To submit your own memory-generation harness too, add an adapter by PR.

How to submit