Paper: 2611.26361 — How Much Should an Agent Remember? Deployment Lessons from Self-Distilled Memory for LLM Agents
Authors: K. R. Jayaram, Vatche Isahagian, Aswath Vaithinathan Aravindan, Vinod Muthusamy, Punleuk Oum, Gaodan Fang, Michal Ulewicz, Gegi Thomas, Evelyn Duesterwald
Category: cs.AI
Reading scope: Original arXiv v1 HTML, including the system, measurement setting, seven deployment insights, results tables, and product examples. Figure captions were read; figure images were not visually inspected. Reported experiments and product traces were not independently reproduced.

The Problem

An agent has learned a sensible-looking rule for reading a paginated API: keep requesting pages until one returns fewer items than the page limit. It assumes the limit is 20. The API actually defaults to five, so the first page looks like the last. A reminder meant to prevent mistakes now makes the agent miss every later page.

This happened in the study. The rule had emerged repeatedly from earlier tasks, which gave it a high support count. Repetition did not make it correct. The researchers fixed it by hand: pass an explicit limit and continue until a page is empty. That task recovered (Insights, Insight 4).

K. R. Jayaram and colleagues built a system that turns an agent’s past runs into reusable guidance. Their paper contributes deployment lessons about when that guidance helps, how much to deliver, and what can silently go wrong. Adding a memory store is the beginning of that work.

The Explanation

Save experience without filling every slot with the same lesson

The system, altk-evolve, first runs an agent on training and development tasks. The same model that operates the agent reads its own successful and failed trajectories and writes one to three short lessons per episode. Success comes from the evaluator’s verdict, not the model’s account of how well it did (The System).

Five tasks may all produce a pagination reminder. If those reminders occupy the first five retrieval slots, other useful guidance cannot get in. The system therefore groups similar lessons and merges only those whose instructions are equivalent. Each surviving guideline keeps its source information and a count of the lessons it represents. Uncovered lessons are carried forward rather than dropped. For DeepSeek-V3.2, 290 lessons become 191 guidelines, including 144 supported by just one lesson.

The store can retain that long tail while the agent reads only part of it. One delivery mode injects every guideline. Another always includes a core supported by at least three lessons, then retrieves additional guidelines using similarity between their source tasks and the new task. The distinction is practical: preserve the unusual pagination case in storage, then decide whether this task needs it.

The same delivery policy does not suit every model

AppWorld tests whether agents can write and execute Python to operate simulated applications. Task completion means passing every check for an individual task. Scenario completion is stricter: every task variant within a scenario must pass. Each model in this study mined its own memory; the comparisons do not give all models one identical guideline set (Measurement Setting).

On the 168-task normal test split, DeepSeek-V3.2 rose from 79.8% task completion without memory to 89.3% with all 191 guidelines. Scenario completion rose from 64.3% to 80.4%. A compressed twelve-guideline checklist gave only a 2.9-point task-completion gain, against 9.5 points for the full set. Tidying away situational lessons discarded useful information (Insights 1–2; Table 2).

For gpt-oss-120b, the full set still helped: 39.9% to 51.2% task completion with its 199 guidelines. But selecting about 29 guidelines reached 55.4%, the mean of two runs, 56.0% and 54.8%. The paper’s reported scenario score for that configuration, 37.5%, comes from the first run only. Reading fewer guidelines also kept total token use close to the no-memory baseline: 116,000 versus 110,000 per task, compared with 166,000 for its full set (Tables 2–3).

These are measured differences among tested configurations, not a universal rule that a larger model should receive more memory. GLM-5 did not improve with its full set, and Gemma-4-31B changed little overall. The authors expose the delivery dose as a setting because one policy did not work equally well across the models they tested.

The pagination failure shows why selection cannot rely on support alone. Support counts independent derivations, not correctness. The study also found that consolidation could silently drop the very provenance needed for later selection. The implementation now checks that total support survives merging and carries omitted lessons through unchanged (Insights 3–4).

Even a sound lesson is useless if it never reaches the store. In an early product integration, one code path never called the memory writer. The system could read memory, but accumulated none from new runs. Checking that the store actually grows was more revealing than polishing the retrieval algorithm (Insight 7; Deployment in IBM’s Products).

Core Concepts

Storage and delivery have different budgets. Keeping a rare lesson does not require adding it to every model call. Consolidation removes equivalent duplicates; selection decides which remaining instructions the agent sees now.

Support needs provenance. A count is useful only if its source lessons survive merging accurately. Even then, a high count can represent a common misconception, as the pagination rule did.

The agent scaffold is part of the comparison. Prompts, worked examples, and execution loops affect the no-memory baseline. Comparing final scores across different scaffolds can credit memory for an advantage it did not create (Insight 6).

The Insight

Keep the experience that might matter later; measure how much of it the agent can use now. Compressing the archive and limiting the reading load solve different problems.

A Critical Reading

The study is unusually direct about measurement limits. Because development tasks were used to mine memory, no separate split remained for choosing delivery settings. The reported best configurations were selected on the test split itself. The fixed, prechosen policy of injecting all guidelines is therefore a useful companion to the best-on-test column. The selected 29-guideline result is promising, but is not a validated optimum for unseen tasks (Measurement Setting; Table 2).

Most results are single runs. Identical configurations could change individual task outcomes even with deterministic-looking settings; the authors treat differences below roughly five tasks as noise in these experiments. That is an empirical caution here, not a general significance threshold. Their comparison with ACE illustrates the consequence: gpt-oss accuracy was effectively tied, while token use differed substantially. DeepSeek’s final-score gap also shrank when measured as improvement over each scaffold’s own baseline (Insights 5–6).

The paper reports deployment in IBM products, but its expense-claim trace compares different inputs. It illustrates transfer of a procedure, not a controlled latency improvement. Nor does the system yet automatically establish whether guidelines are contradicted, stale, or harmful. The pagination repair required human intervention.

What You Can Use

For an agent that repeatedly mishandles APIs, start with a small collection of actual failed and successful runs. Preserve source tasks when merging equivalent lessons. Then compare no memory, the full set, and a selected subset on the same agent setup, measuring completion and tokens together. Repeat close results. Finally, read the highest-support rules against the API contract: a well-remembered wrong instruction can be more damaging than an absent one.

论文:2611.26361 — How Much Should an Agent Remember? Deployment Lessons from Self-Distilled Memory for LLM Agents
作者:K. R. Jayaram、Vatche Isahagian、Aswath Vaithinathan Aravindan、Vinod Muthusamy、Punleuk Oum、Gaodan Fang、Michal Ulewicz、Gegi Thomas、Evelyn Duesterwald
分类:cs.AI
阅读范围:阅读 arXiv v1 原始 HTML 中的系统、测量设置、七项部署经验、结果表和产品案例。读了图注,未查看图像本身;未独立复现实验或产品操作记录。

问题

一个智能体记住了翻页的诀窍:每次取一页,拿到的条目少于每页上限,就停。它以为上限是 20,接口默认却只给 5 条。第一页拿到 5 条,它当场收工,后面的页一条没读。

这不是编的例子。论文里的这条经验来自多次任务,支持它的记录还不少。可重复出现,不等于说得对。研究者最后手工改写规则:请求时明确指定每页数量,一直翻到空页再停。那道原本做错的任务恢复了,见「经验」第 4 项。

K. R. Jayaram 等人搭了一个系统,从智能体自己的操作记录里提炼经验,再供它以后使用。这篇论文的贡献主要是部署所得:经验什么时候有用、一次该给多少,以及流程会在哪里悄悄失灵。存进记忆库,只是开头。

翻译

留住经验,别让重复提醒挤满位置

系统叫 altk-evolve。智能体先做训练集和开发集任务,再由同一个模型回读自己的成功与失败记录,每次提炼一到三条短经验。任务成没成功,要看评测器的判定,不能听模型自己报喜,见「系统」一节。

五个任务都踩过翻页的坑,就可能留下五条类似提醒。检索时,前五个位置全让它们占去,别的经验便挤不进来。因此,系统把相近经验聚在一起,只合并执行含义等价的条目。每条留下的指南保留来源,另记自己合并了几条经验。模型没覆盖到的原条目也要原样带过去。DeepSeek-V3.2 的 290 条经验就这样归并成 191 条指南,其中 144 条只得到一条经验的支持。

库里可以留得全,临场未必全读。一种投放方式把所有指南塞进上下文;另一种先放入至少得到三条经验支持的核心,再按新旧任务的相似度检索补充。翻页的罕见情况可以先留在库里,到了相关任务才拿出来。

同一剂量,换个模型就未必合适

AppWorld 让智能体写 Python,操作模拟的邮件、支付、购物等应用。任务完成率看单项任务是否通过全部检查;场景完成率更严,同一场景里的所有任务变体都得过。这里每个模型都从自己的记录里提炼记忆,并非所有模型共用一套指南,见「测量设置」。

在 168 道普通测试任务上,DeepSeek-V3.2 从不带记忆的 79.8% 任务完成率,升到读入全部 191 条指南的 89.3%;场景完成率则从 64.3% 升到 80.4%。如果先压成一张十二条的整洁清单,任务完成率只涨 2.9 个百分点,完整集合则涨了 9.5 个点。那些只遇见过一次的情境经验,并没有因为少见就失去价值,见经验第 1—2 项及表 2。

gpt-oss-120b 也受益,但曲线不同。不带记忆时任务完成率为 39.9%,读完自己的 199 条指南后到 51.2%;只选约 29 条,反而达到 55.4%。最后这个数是两次运行 56.0% 和 54.8% 的均值。论文所列该配置 37.5% 的场景完成率,只来自第一次运行。少读也省开销:每任务总 token 从无记忆的 11 万变成约 11.6 万,全量投放则要 16.6 万,见表 2—3。

这些是所测配置之间的差别,推不出「模型越大,就该记得越多」。GLM-5 读完自己的完整集合后没有提高,Gemma-4-31B 整体变化也很小。作者把投放剂量做成配置项,原因正是不同模型吃不下同一份菜单。

别把常见错误养成操作规程

开头的翻页故障提醒人:支持数只能说明经验推导出来多少次,不能判定它是否正确。另一个故障更早发生:归并时,流程曾悄悄丢掉后续筛选需要的来源字段。现在实现会检查合并前后的支持总数是否守恒,遗漏的条目必须保留,见经验第 3—4 项。

规则再好,没写进库也没用。早期产品集成里,有一条代码路径从未调用记忆写入器。系统表面上有检索功能,却始终学不到新任务的经验。检查库里的条目有没有增长,比继续琢磨检索算法更能找出问题,见经验第 7 项和「IBM 产品部署」。

核心概念

保存与投放,是两笔预算。 留住一条罕见经验,不等于每次调用模型都要读它。归并去掉含义相同的重复,选择则决定此刻给智能体看什么。

支持数要带着来源读。 来源丢了、重复计数了,筛选依据就坏了。即便计数没错,它也可能代表一个反复出现的误解,翻页规则就是例子。

智能体外壳也影响成绩。 提示词、示范和执行循环都会改变无记忆时的基线。换一套外壳再比最终分数,可能把外壳的功劳算给记忆,见经验第 6 项。

洞见

把以后可能有用的经验留下,再测清楚眼下能用多少。压缩档案与减轻临场阅读负担,是两件事。

博导审稿

论文把测量边界交代得很直白。开发集已用于挖掘记忆,作者没有另外留下选择投放配置的验证集,而是直接在测试集上挑最佳方案。因此,表 2 同时放上了事先定好的「全部投放」规则。约 29 条的检索结果值得继续测,却还不是经独立数据验证的最优剂量,见「测量设置」和表 2。

大部分结果只跑了一次。哪怕设置看似确定,同一配置仍会改变若干任务的成败。作者在这组实验里把不足约五道题的差距当成噪声看待;这不是通用的显著性门槛。与 ACE 的比较说明了原因:gpt-oss 的准确率差距基本打平,token 成本差别却很大;DeepSeek 的最终分数差距,在分别减去各自外壳的基线后也缩小了,见经验第 5—6 项。

部署部分有真实产品经验,但报销案例的前后输入不同,只能说明操作规程迁移到新任务,不能当作受控的延迟提升实验。系统目前也不会自动判定指南是否过时、遭到反例推翻或正在造成损害。开头那条翻页规则,仍靠人改正。

启发

如果智能体反复用错接口,可以先收集一小批真实的成功与失败记录。合并同义经验时保留来源任务,再用同一套智能体分别测无记忆、全量、按任务选择三种方式,把完成率与 token 一起记下来。小差距多跑几次。最后,把支持数最高的规则拿去对照接口契约:一条记得很牢的错指令,可能比没记住更难办。