Benchmarks as Software: Terminal-Bench 2.1 and the Reality of Benchmark Maintenance Debt

Paper: 2611.08417
Authors: E. Kelly Buchanan, Crystal Zhou, Scott W. Linderman, Christopher Ré (Stanford University, Cornell University)
Categories: cs.AI, cs.SE, cs.CL

The Gap

Autonomous coding agents (e.g., Claude Code, Codex, SWE-agent) represent the most economically consequential commercial deployments of frontier LLMs. Consequently, public agentic benchmarks—most notably Terminal-Bench—directly dictate enterprise purchasing decisions, VC funding, and foundational research priorities.

However, the AI community has historically treated evaluation suites as static, immutable datasets. In reality, agentic benchmarks are complex, multi-tiered software systems comprising Docker containers, dynamic network environments, grading oracles, test scripts, and shell execution budgets.

This static mindset causes an invisible, compounding crisis: Benchmark Maintenance Debt.

1. Instruction-Verification Mismatch: As frontier models improve, they find novel, perfectly valid technical solutions. However, rigid assertion scripts (e.g., exact regex matches on terminal stdout or byte-for-byte binary diffs) reject correct completions.
2. Task Bit Rot and Dependency Drift: Tasks that pull live apt packages, pip dependencies, external GitHub repositories, or REST APIs silently break when upstream software versions update.
3. The Hidden Resource Trap: Benchmarks enforce arbitrary wall-clock execution limits. Slow execution environments or rate limits trigger artificial timeouts, punishing thorough models that take seconds longer to execute code.

                    THE BENCHMARK MAINTENANCE DEBT LIFECYCLE

  Task Design:                Agent Execution:              Verification:
  [ instruction.md ]  ───>    [ Docker Runtime $ _ ] ───>   [ Verifier & Oracle ]
  - Underspecified goals      - CPU/RAM limits throttled    - Overspecified unit tests
  - Hidden constraints        - Upstream APIs drift         - Rejects valid novel paths
  - Rigid assumptions         - Wall-clock timeout hit      - Flawed ground-truth oracles
                                        │
                                        ▼
  Result: Over 30% of benchmark tasks develop defects within months of release!

---

The Increment

One sentence: Stanford researchers Christopher Ré, Scott Linderman, and colleagues formalize Benchmarks as Software, conducting a systematic audit of Terminal-Bench 2.0 that uncovered defects across ~30% of its tasks, releasing Terminal-Bench 2.1 with repairs that swing agent scores by over 10 percentage points, while proving that wall-clock time budgets alone explain up to 30 percentage points of apparent harness performance gaps.

                  CONTINUOUS BENCHMARK VALIDATION PIPELINE

  [ Community Bug Reports ] + [ Execution Traces ] + [ Automated Oracle Checks ]
                                     │
                                     ▼
  ┌────────────────────────────────────────────────────────────────────────┐
  │   Evidence Collector Agent: Gathers Docker logs, diffs, & metrics      │
  └────────────────────────────────────────────────────────────────────────┘
                                     │
                                     ▼
  ┌────────────────────────────────────────────────────────────────────────┐
  │   Multi-Model Auditor (Claude Code + Codex): Triages into Taxonomies   │
  │   - Instruction-Verification Mismatch (57% of defects)                 │
  │   - Task Drift (43% of defects)                                        │
  │   - Environment & Resource Issues (29% of defects)                     │
  └────────────────────────────────────────────────────────────────────────┘
                                     │
                                     ▼
  ┌────────────────────────────────────────────────────────────────────────┐
  │   Human-Gated Repairs: Oracle regression tests + TB 2.1 Release         │
  └────────────────────────────────────────────────────────────────────────┘

---

Core Framework & Mechanisms

The authors establish a continuous validation and maintenance pipeline that treats benchmark suites identically to mission-critical open-source software:

1. The Three Primary Defect Taxonomies

Applying this rubric to the 28 repaired tasks in Terminal-Bench 2.0 revealed three distinct failure surfaces:

• Instruction-Verification Mismatch (57% of defective tasks): The problem prompt and the grading assertion script disagree. For example, in the query-optimize task, the verifier script demanded byte-for-byte preservation of SQLite database file headers—a hidden requirement never mentioned in the prompt.
• Task Drift (43% of defective tasks): Breaking changes in mutable external state, such as updated pip wheels, modified third-party API payloads, or changed public documentation. The repair pins all external environments hermetically.
• Environment & Resource Issues (29% of defective tasks): Tasks that are solvable in theory, but fail because Docker containers run out of memory, disk inodes, or CPU time during grading.

2. Auditing Workflow & Multi-Agent Consistency

The validation architecture operates across three distinct roles:

1. Evidence Collector: Extracts stdout, stderr, process metrics, and git diffs from successful and failed model runs.
2. Auditor Agent: Classifies defects against standardized rubrics. Highly capable models (Claude Code with Opus 4.6 and Codex with GPT-5.4) achieve strong inter-agent agreement ($\kappa = 0.80$).
3. Verifier Agent: Re-runs ground-truth reference oracles against proposed patch diffs to ensure the repair fixes the root cause rather than just relaxing test thresholds.

3. The Resource Allocation Confounder

Benchmarks measure a composite product of $\text{Model} \times \text{Harness} \times \text{Resource Budget}$. The authors show that default wall-clock timeouts create severe evaluation illusions:

• On GPT 5.4, relaxing the strict time budget closed a massive 20 percentage-point harness gap between rival agent harnesses.
• On GPT 5.4 mini, extending execution time collapsed a 30 percentage-point gap down to just 5 percentage points.
• Therefore, reported “agent framework superiority” is frequently nothing more than faster script execution under tight timeouts rather than superior reasoning.

---

Key Quantitative Results

The team evaluated 19 state-of-the-art agent setups across Terminal-Bench 2.0 and the newly repaired Terminal-Bench 2.1.

1. Benchmark Audit Impact on Agent Leaderboards

Agent Configuration (Model + Harness)	Terminal-Bench 2.0 Score	Terminal-Bench 2.1 Score	Absolute Accuracy Delta	Leaderboard Rank Shift
Claude Opus 4.6 + Claude Code	80.2%	87.6%	+7.4 pp	Climbed +3 ranks
GPT 5.5 + Codex	88.5%	93.1%	+4.6 pp	Stable (#1)
GPT 5.4 + Codex	82.0%	85.2%	+3.2 pp	Down -1 rank
Gemini Pro 3.1 + Gemini CLI	76.4%	81.0%	+4.6 pp	Stable
GLM 5.1 + Terminus-2	68.5%	72.8%	+4.3 pp	Stable
Claude Sonnet 4.6 + Claude Code	71.0%	76.5%	+5.5 pp	Climbed +1 rank
GPT 5.4 mini + Codex (Default Timeout)	48.2%	51.0%	+2.8 pp	Harness Gap: 30 pp
GPT 5.4 mini + Codex (Extended Timeout)	72.0%	76.2%	+25.2 pp	Harness Gap: 5 pp

2. Analysis of the Repaired Tasks

• Concentrated Lift on Mismatches: Across tasks where verifiers were repaired, 5 tasks that previously had pass rates $\le 15\%$ jumped to $> 50\%$, accounting for 62.7% of the total accuracy change across the repaired subset.
• Rank Shuffling: While global Spearman correlation remains high ($\rho = 0.958$), leaderboard positions shifted by up to three full ranks, invalidating several marketing claims made on TB 2.0.
• Timeout Dominance: In 68% of failed Claude Code trials and 57% of failed trials across all harnesses, tasks failed purely because of wall-clock timeouts, not incorrect code.

---

Practical Implications

1. Benchmarks Must Adopt Semantic Versioning & CI: Static benchmark releases are obsolete. Autonomous coding benchmarks must be maintained as active software projects with semantic versioning (2.0 $\to$ 2.1), regression tests, and public issue trackers.
2. Decouple Timeout Latency from Capability: Benchmark designers must separate algorithmic competence from execution speed. Enforcing rigid wall-clock deadlines measures server hardware and API latency rather than cognitive reasoning.
3. Auditing Verification Scripts: LLM providers must inspect grading assertions. When frontier models score lower on a benchmark after an update, it is frequently because the model found a novel, more concise solution that violated hardcoded assumptions in the test suite.

                     智能体评测全生命周期的“技术债”蔓延

  任务设计阶段：              智能体执行阶段：              最终判题阶段：
  [ 题目指令 instruction.md ] ──> [ Docker 执行环境 $ _ ]  ──> [ 判题脚本与基准答案 ]
  - 未明示的隐藏限制条件       - 容器 CPU/显存/磁盘超限      - 僵化死板的单元测试断言
  - 出题人的主观狭隘假定       - 外部依赖库更新导致报错      - 误杀模型创新探索的有效解
  - 缺乏防御性验证设计         - 偶发卡顿触发物理超时        - 损坏的 Ground-Truth Oracle
                                        │
                                        ▼
  恶果：即便发布前经过严格人工与红队审查，短短数月内近 30% 的评测任务均出现致命缺陷！

                    智能体基准持续验证与自动化修复流水线

  [ 社区缺陷 Issue 报告 ] + [ 历史运行轨迹 Trace ] + [ 确定性 Oracle 自动化回归 ]
                                     │
                                     ▼
  ┌────────────────────────────────────────────────────────────────────────┐
  │   证据收集智能体 (Evidence Collector)：自动归集容器日志、Diff 与指标   │
  └────────────────────────────────────────────────────────────────────────┘
                                     │
                                     ▼
  ┌────────────────────────────────────────────────────────────────────────┐
  │   多模型审查仲裁 (Auditor Agent: Claude Code + Codex 双核校验)         │
  │   - 归类缺陷类型：指令断言脱节 (57%)、环境漂移 (43%)、资源瓶颈 (29%)   │
  │   - 自动生成候选 Patch 补丁与回归测试用例                              │
  └────────────────────────────────────────────────────────────────────────┘
                                     │
                                     ▼
  ┌────────────────────────────────────────────────────────────────────────┐
  │   人类终审验证与版本迭代：发布 Terminal-Bench 2.1 (消除维护技术债)      │
  └────────────────────────────────────────────────────────────────────────┘