LongMemEval-V2: Teaching Agents to Remember Like Experienced Colleagues

Paper: 2605.12493 Authors: Di Wu, Zixiang Ji, Asmi Kawatkar, Bryan Kwan, Jia-Chen Gu, Nanyun Peng, Kai-Wei Chang Categories: cs.CL

The Gap

Existing agent memory benchmarks focus on user preferences, short interaction traces, or downstream task success metrics. They don’t directly test whether memory systems help agents internalize environment-specific knowledge—the kind of tacit expertise a human colleague builds after months of working with a particular system. You know: which buttons are buggy, which workflows are reliable, what state transitions to expect.

The gap is evaluation methodology. We have memory systems, but no way to measure if they’re actually capturing the right kind of experience.

Problem: How do we know if memory helps agents become "experienced"?
   |
   v
Assumption: Experience = answering questions about environment internals
   |
   v
Method: Curate 451 questions across 5 memory abilities + long histories
   |
   v
Evidence: RAG gets 48.5%, coding agent gets 72.5% (with high latency)
   |
   v
Conclusion: Benchmark is hard; coding agents work but are slow

The Increment

One sentence: Before this paper, we evaluated memory by task success; after, we can directly test whether agents remember environment-specific gotchas, workflows, and state dynamics.

Core Mechanism

LongMemEval-V2 uses a context gathering formulation. The memory system consumes up to 500 trajectories (115M tokens of interaction history) and returns compact evidence snippets. A downstream QA model uses these snippets to answer questions about the environment.

The benchmark tests five memory abilities: static state recall (what’s on the page?), dynamic state tracking (how did this element change?), workflow knowledge (what’s the standard procedure?), environment gotchas (what breaks and why?), and premise awareness (what assumptions does this task make?).

Two proposed methods: AgentRunbook-R is a RAG system with three knowledge pools—raw observations, extracted events, and strategy notes. AgentRunbook-C stores trajectories as files and invokes a coding agent in a sandbox to write Python scripts that gather evidence by parsing trajectory data.

History Trajectories (up to 500, 115M tokens)
   |
   v
Memory System (RAG or Coding Agent)
   |
   v
Compact Evidence Snippets
   |
   v
QA Model --> Answer
   |
   v
Evaluation: 5 memory abilities x 451 questions

Think of it like training a new hire. The trajectories are the new hire shadowing experienced colleagues for months. The memory system is their note-taking strategy—either organized binders (RAG with knowledge pools) or a personal assistant who can search through all their notes and write custom reports (coding agent). The questions are the pop quiz: “Hey, remember that bug with the submit button? What causes it?” A good memory system means the new hire can answer without re-reading all their notes.

The coding agent approach works because it can write custom parsing logic for each question—like having a research assistant who knows Python. The RAG approach is faster but less flexible—like having pre-indexed flashcards that might not have the exact angle you need.

Key Concepts

• Context gathering formulation: Instead of evaluating memory by whether the agent completes tasks successfully, you evaluate it by whether the memory system can retrieve relevant evidence from long histories. It’s like testing a librarian’s filing system by asking them to find specific passages, not by checking if patrons leave satisfied. This decouples memory quality from task execution, letting you diagnose what the memory system actually learned.

• Environment gotchas: These are recurring failure modes specific to a system—the kind of knowledge you only get from experience. For example: “The search button doesn’t work if you type too fast” or “Clicking ‘Save’ twice creates duplicate entries.” Existing benchmarks don’t test this because they focus on user preferences or general knowledge. But this is exactly what makes an experienced colleague valuable—they know where the landmines are.

• Knowledge pools in RAG: AgentRunbook-R doesn’t just dump everything into one vector database. It maintains three separate pools: raw state observations (what’s literally on screen), events (state transitions and actions), and strategy notes (high-level patterns). This is like organizing a filing cabinet by document type instead of throwing everything in one drawer. When you search, you can target the right pool for the question type.

Framework Shift

Before (mainstream approach):        After (this paper):

Agent --> Task Success               History --> Memory System
  ^           |                        |            |
  |           v                        v            v
Memory    Evaluation                Evidence --> QA --> Eval
(implicit)                          (explicit test of memory)

Focus: Did the agent win?            Focus: What did the agent learn?

From outcome-based to knowledge-based evaluation, the core shift is making memory testable as a first-class capability.

Expert Assessment

Problem choice: This is a real gap. The field has been building memory systems without direct ways to measure what they remember. The focus on environment-specific experience (not user preferences) is smart—it targets the kind of knowledge that’s hardest to acquire and most valuable in specialized domains.

Method maturity: The benchmark design is solid—manually curated questions, diverse memory abilities, realistic history lengths. The two proposed methods are reasonable baselines but not groundbreaking. AgentRunbook-C is essentially “let an LLM write code to parse trajectories,” which is powerful but not novel. The RAG baseline with knowledge pools is sensible but incremental. The paper’s value is the benchmark, not the methods.

Experimental integrity: Baselines are fair. The 72.5% vs 48.5% gap between coding agent and RAG is convincing, but the latency cost (not quantified in the abstract) is a major caveat. The paper acknowledges this honestly. One concern: 451 questions is decent but not huge—there’s risk of overfitting to question types. Would like to see inter-annotator agreement scores for question quality.

Writing quality: The abstract is clear but undersells the benchmark’s design. The “experienced colleagues” framing is good but could be pushed harder—what does it mean operationally? The methods section likely has the most room for improvement: need clearer ablations on what each knowledge pool contributes in AgentRunbook-R, and more analysis of when coding agents fail.

Verdict: weak accept — Solid benchmark contribution with honest evaluation, but the proposed methods are incremental and the dataset size is moderate. The field needs this benchmark, but the paper doesn’t push the solution space hard enough.

Takeaways

Steal the evaluation formulation: Context gathering (memory system returns evidence, separate QA model answers) is a clean way to test retrieval quality without confounding factors. You can apply this to any domain where you want to test if a system “learned” from experience—not just agents, but also recommendation systems, debugging assistants, or code review tools.

Steal the knowledge pool architecture: Separating raw observations, events, and strategy notes in RAG is simple but effective. If you’re building a memory system, don’t dump everything into one vector store—organize by information type and query the right pool.

Steal the “gotchas” category: Testing whether a system remembers recurring failure modes is underrated. If you’re evaluating any learning system in a specialized domain, add questions about edge cases and failure patterns—that’s where experience shows.

Don’t steal the coding agent approach blindly: It works but has high latency. Only use it if accuracy matters more than speed, or if you can cache/precompute evidence for common question types.