Paper: 2611.24924 — Agentic Harness Engineering Beta: Requirement-Centered Evolution of Agent Harnesses Authors: Jiahang Lin, Jiahao Chen, Yiding Wang, Shichun Liu, Chengjun Pan, Shihan Dou, Honglin Guo, Yuyang Hu, Changhao Jiang, Zhenhua Han, Hang Yan, Qi Zhang, Tao Gui, Xuanjing Huang Categories: cs.SE, cs.AI Reading scope: Original arXiv HTML: main method, experiments, limitations, and Appendix F’s human-study protocol. Figures were not visually inspected; results come from text and extracted tables. Experiments were not reproduced.

The Problem

Imagine asking a coding assistant to diagnose an export bug without changing code. It applies a patch. The tests pass, but your request was not followed. This illustrative situation resembles Appendix F’s staged Django task: diagnose, discuss, obtain permission, modify, accommodate follow-ups, and verify.

A checker can inspect the repository and miss whether the assistant respected scope, noticed a correction, or delivered its result. Improving its score does not necessarily improve those behaviors.

AHE-Beta keeps the underlying model fixed and improves its harness: the prompts, tools, and execution logic around it. Instead of selecting revisions by benchmark score, it admits user requirements supported by session evidence and merges implementations that pass review.

The Explanation

Turn a frustrating session into a reusable requirement

The loop starts with a simulated user working with the assistant over several turns. Its persona sets expectations about autonomy and completion evidence. The assistant cannot inspect that profile; it learns through messages. The simulator files issues and rates the experience (§3.1).

In the export example, “fix this query” belongs to the user’s task. A reusable requirement might separate permission to diagnose from permission to edit. Triage checks complaints against logs and code, rejects invalid or task-specific issues, and removes project names and particular answers (§3.2). Missing evidence prompts a requirement for better logging, rather than a guessed cause.

An Architect combines requirements sharing a structural problem. Separately, a Product Manager finds unstated needs in session summaries and ratings. Developers compare designs, write acceptance criteria, implement admitted requirements, and open pull requests. A complaint becomes traceable to a software change.

Ask whether the change solves the requirement

Two gates control this (§3.3). Admission requires session evidence. Review checks design, code, acceptance criteria, and completed tests. Approval must apply to the current pull-request head, rather than an earlier version.

Developers receive runtime logs, public task descriptions, and execution conditions, but not hidden tests, graders, or raw user conversations. Benchmark scores are reserved for evaluation after evolution. The aim is a reusable behavior change, rather than a patch tailored to the current test.

The engineering-team metaphor is literal: AHE-Beta automates requirements, implementation, and review around an unchanged model.

Check the outcome at more than one level

The main evaluation uses GLM-5.3-Flash on 38 held-out Harbor-index tasks, with simulated-user follow-ups. Figure 2 reports means over three evaluation runs, each allowing one attempt per task, rather than a single set of 38 outcomes. Audited one-attempt task success, or pass@1, rises from 21.9% to 33.3% over the starting harness: 11.4 percentage points. Mean user rating rises from 2.59 to 3.04 on a 1–4 scale (§§4.1–4.2; Figure 2 and Appendix E).

Mean execution tokens, the units of text a model processes and generates, fall from 2.40 million to 1.42 million per task, a 40.8% reduction (Table 4). This excludes simulator tokens and the full evolution cost. More efficient execution does not establish a 40.8% saving on the whole improvement process.

Table 2 separately tests whether feedback during a session matters. Two variants evolve from eight scenarios, one allowing multi-turn interaction and the other receiving feedback only at the end. With both then evaluated using interaction, success is 28.9% versus 23.7%. This comparison supports the value of mid-session corrections, rather than attributing every gain to having more engineering roles.

The revealing finding concerns tasks that “passed”: 34 of 137 verifier-passing sessions never delivered a completed reply and received the simulator’s minimum rating (§4.3). The checker read the workspace; the user saw the assistant’s communications. A correct artifact and a delivered outcome were different things.

A blind human study on six coding tasks favors AHE-Beta in 31 of 48 valid pairs, against seven preferences for the starting harness, nine ties, and one “neither” (Table 1; Appendix F). These come from 25 anonymous participant labels, not 48 independent people. The direction agrees with the simulator, within a small study using self-reported task completion.

Core Concepts

Harness versus model. Tool use, work tracking, message handling, and delivery can change without changing model weights. The main comparisons hold the execution model fixed; this is not a general commercial-model ranking.

A requirement versus a task-specific fix. Fixing one export completes a job. Keeping diagnosis separate from editing permission could govern many jobs. This is an illustrative generalization; admission still requires evidence that the problem exists.

Verification versus delivery. A test suite checks the workspace, not whether the user received a usable result. The paper measures task success and experience separately.

The Insight

An agent can produce working code while losing a correction or failing to communicate completion. Improving those behaviors requires interaction evidence alongside artifact checks. “Be more helpful” is hard to test; a concrete requirement about scope or delivery can be linked to a session, implemented, and checked.

A Critical Reading

Evidence extends beyond simulator ratings: held-out success, the comparison with final-only feedback, human preference, and cross-model transfer. Without further evolution, success rises by 15.8 percentage points on GLM-5.3 and 5.3 points on DeepSeek-V4.1-Flash over their own starting harnesses (Table 4). That supports some transfer beyond the original model.

Boundaries remain: 38 held-out tasks, one evolution run per setting, one simulator model, and selected personas (§4.1 and Limitations). Human completion reports received no independent code review (Appendix F). The bundled roles and gates also do not establish that every component is necessary everywhere.

The authors warn that unclear requirements can make an overly proactive harness exceed user permission. An inferred preference must not become assumed authorization; that would recreate the opening problem.

What You Can Use

For the export assistant, record both whether the artifact works and whether the interaction respected the request. Turn observable failures into reusable requirements with falsifiable acceptance criteria. You need not deploy the whole agent team to make a complaint survive that journey. Green tests remain necessary, while scope, steering, and delivery keep their own evidence.

论文: 2611.24924 — Agentic Harness Engineering Beta: Requirement-Centered Evolution of Agent Harnesses 作者: Jiahang Lin, Jiahao Chen, Yiding Wang, Shichun Liu, Chengjun Pan, Shihan Dou, Honglin Guo, Yuyang Hu, Changhao Jiang, Zhenhua Han, Hang Yan, Qi Zhang, Tao Gui, Xuanjing Huang 分类: cs.SE, cs.AI 阅读范围: 阅读原始 arXiv HTML 文本中的主要方法、实验、局限,以及附录 F 的真人试验流程。未对图片做视觉检查,下文数字来自正文与提取的表格;这是论文解读,没有复现实验。

问题

假设你让编程助手先查清报表导出为何出错,暂时别改代码。 它直接补了一段代码,导出正常,测试全绿。 事情办成了吗?程序可能修好了,你的要求却没有得到遵守。 这是一个说明问题的假设场景;论文的真人试验确实包含一项分阶段的 Django 任务:先诊断且不改代码,再讨论方案、授权修改、在执行中追加要求,最后核验结果,见附录 F。

自动检查器可以读最后的代码,却不一定知道用户在途中说过什么。 它也许看不见助手有没有听到纠正、有没有越过修改范围,甚至不知道助手最后是否把结果交给了用户。 因此,把测试得分往上推,不等于把整段合作变好。

AHE-Beta 要改进的正是这层合作。 它不训练底层模型,而是从用户没有得到满足的需求出发,修改模型周围的软件:提示词、工具和执行逻辑。 论文把这层软件叫作 harness,可以把它理解成智能体的工作规程和工具箱。 新版本是否合并,依据是经过核实的需求和代码审查;基准成绩留到演化结束后再测。

翻译

先把一次不满变成一条可复用的要求

每一轮从使用开始。 一个模拟用户带着自己的偏好,与当前版本的助手多轮合作,途中可以报问题,结束后给体验打分。 偏好包括愿意给助手多大自主权、怎样才算看见了完成证据等。 助手不能偷看这份用户画像,只能从消息中理解对方,见第 3.1 节。

回到报表的例子。 “把这个 Django 查询修好”只是当前任务。 “用户只授权诊断时,不要自动进入修改阶段”才可能成为跨任务复用的行为要求。 这条表述是帮助理解的方法示例,不是论文中一条需求的逐字引用。

论文让一个分诊角色先查用户报的问题、运行日志和当时的助手代码。 如果投诉不成立、已经解决,或者只是在要求完成那道题,就不送去修改智能体。 剩下的问题会去掉项目名、题号和具体答案,整理成通用要求。 日志不足以说明原因时,分诊角色应提出“把这件事记录清楚”的需求,而不是猜一个根因,见第 3.2 节。

接着有两条整理路线。 架构师把具有共同根因的要求合在一起,寻找应该改哪一层机制。 产品经理另读会话总结和体验评分,寻找用户没有主动写成投诉、却反复遇到的困难。 开发者拿到获准处理的需求后,比较方案,写出验收条件,再实现为代码和拉取请求。 一次不满由此不再停留在“下次注意”的便签里,而是有机会进入软件本身。

合并之前,先问改动到底解决了什么

第 3.3 节设了两道门。 第一道是需求准入:没有会话证据支撑的要求,不能直接分派开发。 第二道是审查:先看设计能否达到目标、能否被测试推翻,再看代码是否满足验收条件,完整测试是否跑完。 合并批准还必须对应拉取请求当前的代码版本,不能拿旧版本的通过,替新改动背书。

开发者收到的证据包包括运行日志、公开任务说明和执行条件,不包含隐藏测试、评分器或原始用户对话。 每次改动也不靠基准分数决定去留。 这个设计试图把“专门给某一道题打补丁”,改成“修复这类用户都会遇到的行为问题”。

可以把它看作一个把用户投诉接到代码审查的工程团队。 模拟用户负责暴露问题,需求角色负责核实和整理,开发与审查负责把改动落地。 这里的类比并不是额外装饰:这些角色与中间文档就是论文实际采用的协作方式。

然后把做对、用好和花费分开量

主要实验选了 38 项没有用于演化的 Harbor-index 任务,涵盖编程、优化、分析和工具使用。 执行任务的底层模型固定为 GLM-5.3-Flash,主要评测允许模拟用户在助手回复后继续追问或纠正。 图 2 报告的是三轮评测的平均值,每轮每项任务允许尝试一次,因此下面的百分比并非一组 38 次结果的简单计数。 在这个设置下,单次尝试通过任务的比例从起始版本的 21.9% 升到 33.3%,增加 11.4 个百分点。 模拟用户在 1—4 分量表上的平均体验评分从 2.59 升到 3.04,见第 4.1—4.2 节、图 2 及附录 E。

平均每项任务的执行量从 240 万个词元降到 142 万个,减少 40.8%,见表 4。 词元是模型处理和生成文字时使用的计量单位。 这里没有计入模拟用户的用量,也不是整个演化过程的总账单。 能据此说的是:在这套任务执行设置里,改过的助手用了更少的执行词元;不能把它改写成“整个研发成本下降四成”。

表 2 另外检查了反馈出现的时机。 两个版本都从八个场景演化,一个允许多轮交互,另一个只在会话结束后接收反馈。 之后都用允许交互的协议评测,通过率分别为 28.9% 和 23.7%。 这项比较支持中途纠正本身有用,不能把所有收益笼统归给工程角色变多了。

最值得停一下的,却是一组“已经通过”的结果。 第 4.3 节指出,在其讨论的评测会话里,137 次通过检查的会话中,有 34 次没有交出完整的最终回复,模拟用户因此给了最低体验分。 检查器读的是最终工作区,用户看见的是助手向自己交付了什么。 文件在磁盘上做对了,用户却还在等一个结果,这两件事可以同时发生。

作者另做了真人盲测,检查模拟用户喜欢的改动,真人是否也认可。 六项编程任务产生 48 组有效对照,其中 31 组偏好 AHE-Beta,7 组偏好起始版本,9 组平局,1 组两边都不选,见表 1 与附录 F。 这 48 组来自 25 个匿名参与者标识,不能说成 48 位彼此独立的用户。 它支持两种评价在方向上一致,但还不是大规模用户研究。

核心概念

工作规程与底层模型。 同一个模型,外面的工具、状态记录和执行规则不同,行为就可能不同。 在报表例子里,区分“正在诊断”和“获准修改”可以是一条工作规则,不一定需要重新训练模型。 论文的主要对照固定执行模型,因此也不能把结果说成对商业模型能力的普遍排名。

通用需求与当前任务。 修好某一次导出,是当前任务;以后遇到只准诊断的请求,都能守住范围,才是可复用的改进。 后者仍然要用会话证据证明问题存在,不能因为听起来合理,就默认用户已经授权。

核验与交付。 测试能证明工作区满足某些条件,不能自动证明用户知道结果、能使用结果,或者中途的约束得到了遵守。 把两者分开量,才看得见“测试绿了,但用户还没收到回复”这种失败。

洞见

智能体的失败,不全是解不出题。 有些失败发生在它辨认、保存和交付用户真正授权的那件事时。 要改进这些行为,就得把交互过程留下的证据,与最终文件的检查结果放在一起看。

论文真正可取的一步,是让反馈变成可追溯的要求。 “再贴心一点”无从验收;范围、纠正与交付上的具体行为,才可能对应一段会话、一处改动和一次核验。

博导审稿

这项工作的证据没有停在模拟用户给高分。 它还检查了留出任务、中途反馈与仅结束后反馈的对照、真人偏好,以及改动换到底层模型后是否仍有收益。 表 4 中,同一套演化后的工作规程不再继续改动,迁到 GLM-5.3 和 DeepSeek-V4.1-Flash 后,相对各自起始版本的通过率分别增加 15.8 和 5.3 个百分点。 这支持改动具有一定迁移性,而非只对演化时那个模型有效。

但范围仍然清楚。 第 4.1 节与局限部分说明,每种设置只做了一次演化,使用一个模拟用户模型和选定的用户画像,留出任务也只有 38 项。 附录 F 的真人完成情况由参与者自行报告,没有独立代码审查。 这套系统又同时包含多个需求角色与审查门槛,不能只凭总体收益,就断言每个角色在所有部署里都必不可少。

还有一个必须保留的前提:更主动,并不天然更好。 作者在局限部分明确提醒,需求不清时,过度主动的助手可能超出用户许可。 如果系统把“我推断你大概想要”当成“你已经授权”,它只是绕了一大圈,又回到了开头的问题。

启发

如果你维护一个报表助手,可以同时留下两份记录:产物是否正确,过程是否守住用户要求。 会话出问题时,先记录看得见的行为,再写出希望复用的规则,最后给拟议改动一个可能失败的验收条件。

不必一开始就部署论文里的整支智能体团队。 先让一条用户投诉完整走到验收条件,这件小事就有用。 测试全绿仍然必要;范围、纠正和交付,也应该有自己的证据。