An Empirical Study of Harness Design for Coding Agents

Paper: 2609.20804 Authors: Run-Ze Fan, Zihao Zhang, Simin Ma, Yebowen Hu, Shouju Wang Categories: cs.AI, cs.CL, cs.LG, cs.SE

The Gap

Autonomous coding agents have transitioned from curiosity to production infrastructure. Benchmark scoreboards like SWE-Bench Verified and Terminal-Bench attribute performance gains almost exclusively to base model capability. Yet in practice, models do not touch repositories raw; they run inside an execution harness.

The problem is that existing harness evaluations treat the scaffold as a monolithic black box. When OpenHands, SWE-agent, or Claude Code achieves high resolution rates, it remains unclear whether the win comes from superior tool interfaces, elaborate planning loops, or aggressive context compression. Designers routinely stack features—recoverable context caches, hierarchical planning, specialized editing tools—without measuring individual component contributions.

   PROBLEM: MONOLITHIC HARNESS EVALUATION

   Base Model (e.g. Claude / Qwen / DeepSeek)
                       |
                       v
   +-------------------------------------------------+
   | Monolithic Coding Harness                       |
   | [Planning?] + [Bash vs Custom Tools?] + [Cache?]| -> SWE-Bench Score
   +-------------------------------------------------+
                       |
   Gaps: Which component actually drives accuracy?
         Which components add overhead without gain?
                       |
                       v
   METHOD: CONTROLLED 3-COMPONENT FACTORIZATION
   Fixed Execution Loop + 176 Matched Settings
   [Planning (Off/On)] x [Action Space] x [Context Management]
                       |
                       v
   EVIDENCE: 4 Models on SWE-Bench Verified & Terminal-Bench 2.1
                       |
                       v
   CONCLUSION:
   - Rule elision > LLM summarization
   - Recoverable elision adds machinery models never use
   - Planning scaffolds weak models, saves cost on strong models
   - Bash-only interface wins for bash-proficient models

The Increment

One sentence: By holding the agent execution loop fixed and systematically isolating planning, action space, and context management across 176 matched settings, this paper demonstrates that harness components have distinct, decoupled behavioral levers: context management extends trajectory length, planning governs stopping conditions, and action spaces set editing granularity.

Core Mechanism

The study introduces a modular, lightweight harness designed for controlled component ablation. Rather than comparing disparate open-source frameworks whose loops differ in subtle ways, the authors fix the core interaction cycle: environment observation $\to$ model response $\to$ tool execution $\to$ environment feedback.

The authors vary three isolated dimensions across four LLMs:

1. Context Management: Five distinct policies tested across 4 context-window budgets (16K, 32K, 64K, 128K).
   • Sliding window: drops oldest turns.
   • Rule-based elision: prunes verbose command outputs (compiler dumps, long git diffs) via heuristics.
   • LLM summarization: condenses previous interaction turns into prose summaries.
   • Staged elision-then-summarization: rule elision first, followed by summarization only when budget remains tight.
   • Recoverable elision: saves pruned text into numbered files with tool access to retrieve them.
2. Action Space: Contrasting predefined programmatic primitives (dedicated view_file, edit_lines, create_file APIs) against a raw, bash-only execution interface (cat, sed, python, git).
3. Planning Scaffold: Explicit upfront and dynamic step-by-step milestone planning versus zero-planning direct action.

   HARNESS DECOMPOSITION ARCHITECTURE

   Agent Interaction Turn
              |
              v
   +----------------------+
   |  Context Management  | -> Extends execution horizon
   |  (Rule > Summary)    |    (Prevents context overflow)
   +----------------------+
              |
              v
   +----------------------+
   |  Planning Scaffold   | -> Controls trajectory termination
   |  (Scaffold vs Saver) |    (Prevents endless looping)
   +----------------------+
              |
              v
   +----------------------+
   |  Action Space        | -> Dictates editing granularity
   |  (Bash vs Custom)    |    (Lower token overhead for bash)
   +----------------------+
              |
              v
   Environment Execution (Shell / Repo / Test Suite)

The load-bearing structural metaphor is scaffolding around a construction site. The base model is the stonemason.

• Context management is the waste chute: if trash piles up, the mason trips and falls off the platform (overflow failure). Throwing out bulky drywall offcuts (rule-based elision) immediately clears the floor; cataloging every scrap into labeled bins in case the mason needs it later (recoverable elision) takes extra space and the mason never goes digging through the scrap bin.
• Planning is the structural blueprint: an apprentice mason needs the blueprint pinned to every beam to avoid building in circles (accuracy scaffold); a master mason already knows the floor plan, so the blueprint simply reminds them when the job is done and prevents them from over-polishing (stopping condition & token cost saver).
• Action space is the toolbelt: novice masons benefit from pre-shaped molding trowels (specialized tools); experienced masons prefer a single versatile hammer and chisel (bash shell), doing the same work faster with fewer tool changes.

Key Concepts

• Rule-based Elision vs. Summarization: Truncating predictable high-entropy outputs (e.g. trimming terminal outputs longer than 50 lines or stripping unchanged file regions) costs zero inference tokens and preserves syntax. LLM summarization consumes tokens, introduces hallucination risks, and washes out exact compiler line numbers.
• Recoverable Elision Overhead: Theoretical harness designs often give agents retrieval tools to fetch compressed context on demand. In practice, models almost never invoke these retrieval tools during multi-turn problem-solving, making the retrieval machinery pure deadweight.
• Trajectory Lever Decomposition: Harness components affect execution orthogonally: context management prevents premature exit due to window exhaustion; planning dictates termination threshold; tool choice affects token payload per turn.

Framework Shift

Before (Monolithic Harness Assumption):
  More Scaffolding = Better Performance
  +--------------------------------------------------------------+
  | Add complex planning + recoverable context trees + 15 tools |
  +--------------------------------------------------------------+
  -> High token cost, unpredictable interaction effects, debugging nightmare

After (Decoupled Modular Harness):
  Component-Specific Matching by Model Capability & Context Budget
  - Context tight? -> Staged rule-based elision (zero-cost, syntax intact)
  - Capable model? -> Bash-only action space (lower token footprint)
  - Frontier model? -> Lightweight planning solely to bound trajectory cost

From “build the most elaborate agent scaffold possible,” the core shift is that harness features must be selected based on base-model capability and context budget, not accumulated as a virtue.

Expert Assessment

Problem choice: Outstanding. As frontier models converge on benchmark capabilities, engineering leverage has decisively shifted to runtime scaffolding. Dissecting the harness with rigorous controls addresses a real operational blind spot.

Method maturity: The experimental design is disciplined. Fixing the interaction loop while manipulating three orthogonal axes across 176 matched configurations eliminates the confounders that plague cross-framework comparisons. The observation that recoverable elision goes unused is an especially valuable negative result that spares engineers wasted implementation effort.

Experimental integrity: Tested across four models on both SWE-Bench Verified (code editing and test passing) and Terminal-Bench 2.1 (CLI tasks). The ablation on budget-constrained contexts (16K to 128K) is thorough and realistic. Baselines are consistent, and trajectory-level metrics (length, cost, failure modes) support the accuracy claims.

Writing quality: Direct, empirical, and free of hype. The breakdown between accuracy-oriented and cost-oriented benefits provides clear decision boundaries for system builders.

Verdict: strong accept — A definitive empirical benchmark on agent scaffolding that separates cargo-cult design from genuine runtime leverage.

Takeaways

• Ditch recoverable context retrieval in coding agents. Use deterministic rule-based truncation of command outputs before attempting LLM summarization.
• For top-tier models (Claude 3.5 Sonnet class), prefer a raw bash execution environment over extensive custom tool APIs; it reduces token overhead and improves velocity.
• Use planning prompts not to make capable models smarter, but to force them to stop early and verify, cutting run costs.