FrontierMemory 1.0

Today we are excited to release FrontierMemory 1.0, a new benchmark and way to evaluate agent memory through improvement on downstream tasks.

In the past year, memory and continually learning AI systems have broken into the mainstream, with nearly all major coding and personal agents supporting some form of file-based self-editing memory (Instinct, Muse) and even adapting the harness itself (Prime Agent’s “continual harness”). Many agent products (Claude Agents SDK, Letta Code, Cognition’s Devin) support dreaming that adapts memory by studying past trajectories. More generally, scaling sleep-time compute is a key axis for improving agent capabilities (e.g. Harvey’s recent work on wake-sleep agents).

Despite the pervasiveness of agents that claim to learn over time, existing memory benchmarks focus on using agents’ past experience purely for recall, not improvement. Benchmarks such as LoCoMo and LongMemEval evaluate memory by quizzing agents on personal facts like birthdays or hobbies. With recent model improvements and coding harnesses with bash-based file search, these benchmarks have largely been saturated, and also do not actually measure an agent’s ability to learn and improve from experience.

With FrontierMemory, we evaluate agents’ ability to improve by learning from related experience data (trajectories) for tasks that require memory. We use FrontierMemory to evaluate memory across both models and harnesses that support agent learning.

Building FrontierMemory

Benchmark tasks today are designed for the stateless, single-task setting: agents are provided with all the necessary context to solve a task within the task itself. To evaluate how well agents learn with experience, tasks must be memory dependent - where solving the task requires or benefits from memory derived from prior experience. This allows us to distinguish between agents that require all context to be provided upfront (like a brand new intern) from agents that have learned important context over time (like a senior employee).

In FrontierMemory, we manually adapt challenging stateless tasks from public coding and knowledge-work benchmarks (DeepSWE, SkillsBench, AutomationBench, APEX-Agents, and EnterpriseOps-Gym) into stateful tasks that require memory learned from past experience to complete the task. We categorize the types of memory in three ways:

• Task memory: Agents must learn task-specific contracts or conventions that are underspecified in the final task prompt, such as how to handle omitted files in a Word2Vec pipeline.
• User memory: Agents must infer recurring user preferences from prior feedback, such as formatting and example conventions for a public-facing API changelog.
• Environment memory: Agents must learn non-obvious properties of the workspace or domain-specific tools, such as locating environment files and domain rules for energy grid curtailment calculations.

For each task, we create a “gold memory” that contains all the necessary additional information so that we can validate that the task is solvable with memory.

To evaluate agents learning from experience, we also collect a set of trajectories for agents to learn from, where only a subset (219 / 665) are relevant to the evaluation tasks. We collect trajectories from HarborIndex and DeepSWE, and also run various harnesses (Codex, Claude Code, OpenHands, etc.) on the downstream tasks. To enable learning user memory, we run simulated users to generate trajectories (similar to SWE-Interact). Agents are given access to the full set of trajectories for memory generation.

Each task was manually reviewed by at least two Letta researchers, who checked that:

1. The source trajectories contain enough information to learn the memory the agent needs
2. The memory is non-trivial to learn from the trajectories
3. The task verifier grades only what the task asks for

Across review, we found that 53% of the original sourced tasks we kept had broken verifiers that checked for things the task never asked for, rejected correct answers over formatting, or required things the task's tools didn't allow. We repaired these verifiers for FrontierMemory. In total, FrontierMemory 1.0 has 15 tasks.

Findings

We evaluate different models for memory generation by running the Letta Dream harness to generate memories with seven models (Opus 5.5, GPT-6.1 Sol, GLM 5.3, Kimi K3, Gemini 4 Argon, Muse Wonder, and Grok 4.7). Memories are evaluated by being plugged into an agent running on Letta Code, which works on the benchmark tasks. We run the agent working on the benchmark tasks with Opus 5.5 and GPT-6 Sol at five reasoning effort levels (low to max); the leaderboard above reports the better of the two at max effort.

We also evaluate different harnesses. For each harness, we use the same model for memory generation and for tasks (Opus 5.5 and GPT-6 Sol at max reasoning effort). We also use corresponding harnesses for dreaming and for working on tasks: Letta Dream with Letta Code, Claude Managed Agents Dreaming with Claude Code, OpenAI Agents SDK’s dreaming feature with Codex, Prime Agent (using the built-in “refine” feature), and Hermes Agent (using the built-in “learn” feature).

Qualitative Analysis

Instructions shape agent behavior more effectively than summaries

One dimension of memory quality is how well the generated memory is able to be used by the task agent. Prompts which include instructions on how to use available memory, and skill descriptions with imperative phrases, help the task agent effectively use memory while working on tasks. For example, both Opus 5.5 and GPT-6.1 Sol created skills related to Brightpath Software. Opus 5.5’s skill description includes imperative phrasing that precisely instructs an agent on when to use the skill:

In contrast, GPT-6.1 Sol creates 4 skills for the same task, each with more generic descriptions of the workflows they each contain:

The task agent successfully loaded the relevant skill when running with the Opus 5.5 generated memory, but not with the GPT-6.1 Sol generated memory.

Memory that is overfit to past experience fails to generalize

A common failure case for memory generation is failing to identify general patterns, and instead recording overly specific rules tied to what happened in past runs. For example, in one scenario Opus 5.5 learned that the method for an analysis was sometimes specified only in email, so it needed to be looked up:

Learning from the same trajectories, Gemini 4 Argon noticed that in some runs the team's meeting-notes file was a shorter version and the method was in email instead, so it generated the memory:

Because the rule was keyed to a file size rather than the task, it was never triggered on a new task whose method was in email.

Effective memory transfers across models

We did not find any evidence of model-harness “memory codependence”, where memory generated by a specific model works best for the same model. Opus 5.5 generated memory improved performance the most overall, even when the agent running the task was running on GPT-6 Sol. Overall, the ranking across models was roughly the same regardless of if the model solving the tasks run on Opus 5.5 or GPT-6 Sol.

Future Work

FrontierMemory is an ongoing effort and we will continue to evaluate additional models and memory settings.

In this release, we focused on the dreaming setting, where memory is refined from a set of existing trajectories offline. In future releases, we aim to expand on a couple dimensions. Settings where agents generate memory over longer horizons, and wider scope of memory, i.e. harness modifications. Another common setting is online memory generation, where the agent itself or background subagents make updates to memory incrementally as new trajectories run. We plan to extend FrontierMemory to the online setting in future releases, where memory is updated incrementally.

If you are interested in contributing to FrontierMemory or running the benchmark, please contact [email protected].

We are hiring researchers and engineers to work on memory. Join us.