CordisBench: Can a Model Reason About the Harness It Is Rewriting?

Paper: 2609.01600 Authors: Damien Sileo, Dimitri Kachler Categories: cs.AI, cs.CL

The Gap

Dynamic agent harnesses let a language model change the software that shapes its own execution — add a plugin, reconfigure a component, register or tear down a dependency. That flexibility is the point, and it introduces a reasoning burden that ordinary code generation does not have: a local change propagates through dependencies and cleanup. Editing one plugin can leave another in a state nobody intended, and the consequences only appear when something is torn down.

That is a new reasoning task, and it had no benchmark. It is also a task with an unusual structure: unlike most coding problems, the answer is computable — you can execute the configuration and observe the resulting state. So the interesting question is not only whether models can reason about it, but whether they are reasoning at all where an exact method exists.

   DYNAMIC AGENT HARNESSES

   the model can CHANGE the software that shapes its own
   execution: add a plugin | reconfigure a component |
   register or tear down a dependency
        |
        v
   [NEW REASONING BURDEN] a LOCAL change PROPAGATES
     through dependencies and cleanup
     -> editing one plugin can leave another in an
        unintended state
     -> consequences appear only when something is TORN DOWN
        |
        v
   [GAP] no benchmark for this
        |
        v
   AND: unlike most coding problems, the answer is COMPUTABLE
        -> you can execute the configuration and observe state
        -> so the question is not only "can models reason about it"
           but "are they reasoning where an exact method exists"

The Increment

One sentence: Before this paper, lifecycle reasoning about dynamic agent harnesses was unmeasured; after it, a 1,200-question benchmark shows models degrade as interactions multiply — and a finite reference semantics proves the expensive inference was avoidable on these instances.

Core Mechanism

The benchmark has 1,200 questions and combines a controlled formal setting with programs executed against Cordis, a runtime that manages component dependencies and cleanup. That pairing is the design’s strength: the formal setting makes the questions unambiguous, and the real runtime means the ground truth comes from execution rather than from a specification someone wrote.

Four question types, each probing a different part of the reasoning:

• Identify affected components — the propagation question at its most direct.
• Predict state after a specified teardown order — requiring the model to simulate cleanup rather than just dependency edges.
• Determine which conditions hold under all or some orders — a quantifier over orders, which is where the reasoning gets genuinely hard.
• Choose reconfigurations that succeed when executed — the constructive version, where a wrong answer is a broken system rather than a wrong label.

The evaluation targets three efficiency-oriented models at low reasoning effort, with the number of relevant interactions swept over 2, 4, 8, 16, 24 and 32, and deterministic task-specific scoring. The sweep is the experimental spine: it measures degradation as a function of complexity rather than reporting a single accuracy.

Models usually handle small systems well but grow less reliable as more interactions become relevant — and the degradation is uneven across question types. It is worst when predicting final state and when reasoning across teardown orders. Both are the cases where the model must simulate rather than trace: identifying affected components is graph traversal, while predicting the state after a given teardown order requires modelling what cleanup actually does. And reasoning across orders adds a quantifier on top of that. So the failure concentrates exactly where the required operation is simulation rather than lookup.

Additional inference effort recovers marked gains for some models — a partial remedy, and the cost is quantified: on the 16-interaction subset, one model uses nearly 3,000 reasoning tokens per question at medium effort.

Then the finding that reframes the cost. For these controlled instances, that cost is avoidable: an independent finite reference semantics agrees with Cordis execution on every observation and action outcome used for scoring across all 528 executable questions. A finite reference semantics means the answers are derivable by computation rather than by inference. So the benchmark demonstrates a capability gap and shows that on this class of problems the expensive route is not required — which sharpens what the benchmark is measuring: not whether models can compute the answer, but whether they can do it by reasoning where a cheap exact procedure exists.

   BENCHMARK: 1,200 questions
     combination of:
       a CONTROLLED FORMAL SETTING   (questions unambiguous)
       PROGRAMS EXECUTED AGAINST CORDIS
         a runtime managing component dependencies + cleanup
       -> ground truth comes from EXECUTION, not from
          a specification someone wrote

   FOUR QUESTION TYPES
     identify affected components
     predict state after a SPECIFIED TEARDOWN ORDER
     determine which conditions hold under ALL or SOME orders
     choose reconfigurations that SUCCEED WHEN EXECUTED

   EVALUATION
     three efficiency-oriented models, LOW reasoning effort
     relevant interactions swept: 2, 4, 8, 16, 24, 32
     deterministic, task-specific scoring

   FINDINGS
     small systems: usually fine
     more interactions -> LESS RELIABLE
     degradation is UNEVEN:
       worst when predicting FINAL STATE
       worst when reasoning ACROSS TEARDOWN ORDERS
       <- both require SIMULATION, not graph traversal
     more inference effort recovers marked gains for some models
       COST: ~3,000 reasoning tokens per question at medium effort
             (16-interaction subset, one model)

   AND THE COST IS AVOIDABLE ON THESE INSTANCES
     an independent FINITE REFERENCE SEMANTICS agrees with
     Cordis execution on EVERY observation and action outcome
     used for scoring, across all 528 executable questions
     -> the answers are DERIVABLE BY COMPUTATION

Think of it as asking someone to predict the final layout of a kitchen after cooks have been working in it in a specified order. Listing which cupboards a cook touched is bookkeeping and most people manage it. Predicting what the kitchen looks like after a given order of departures is different: you have to know which cooks clean up after themselves and which leave things where they are. And answering “is there a dirty pan on the stove under every possible order” adds a quantifier on top. The paper finds the failures concentrate in the second and third kinds — and then makes the sharper point: because the kitchen has exact rules, you could just read the state off the rules instead of imagining it, so the inference effort is optional for this class of question.

Key Concepts

• Lifecycle reasoning as a distinct task: propagation through dependencies and cleanup, where consequences surface at teardown. It is not ordinary code generation, because the effect of a local edit is not local.
• Execution as ground truth: running programs against a real runtime rather than checking against a written spec. It removes the class of disagreements that come from specification drift.
• Uneven degradation across question types: worst where the required operation is simulation (final state, cross-order reasoning) rather than traversal. It tells you which capability is missing, not merely that accuracy falls.
• Finite reference semantics as the alternative: an exact procedure agreeing with execution on all scored outcomes. It shows the inference was avoidable here, which reframes the benchmark as measuring reasoning where computation suffices.
• Cost as a first-class measurement: nearly 3,000 reasoning tokens per question at medium effort. Reporting the price of the partial remedy is what makes the avoidability finding actionable.

Framework Shift

Before (no benchmark; lifecycle reasoning unmeasured):
  dynamic harnesses let models edit the software that
  shapes their own execution
  -> propagation through dependencies and cleanup unmeasured
  -> no way to say which part of the reasoning is hard

After (1,200 questions, swept by complexity):
  identify affected components | predict state after an order |
  conditions under all/some orders | choose a working config
  -> small systems fine; reliability falls as interactions grow
  -> worst at final-state prediction and cross-order reasoning
  -> and an exact finite semantics matches execution on every
     scored outcome, so the inference was avoidable

From having no way to characterise a new kind of reasoning burden, to knowing which question types fail and that an exact procedure exists for them, the core shift is that the benchmark measures whether models simulate correctly where simulation is not strictly necessary.

Expert Assessment

Problem choice: Excellent, and the timing is right. Dynamic harnesses are becoming the normal way agents are extended, so reasoning about propagation and cleanup is about to be a routine requirement rather than an exotic one — and it is a burden the agent creates for itself, which makes measuring it directly relevant.

Method maturity: The design is well matched to the problem: formal setting for unambiguity, real runtime for credible ground truth, four question types that separate traversal from simulation from quantification, and a complexity sweep rather than a single accuracy. The finite reference semantics is the most sophisticated element, because building and validating an exact alternative to the learned approach is what turns the experiment into a statement about the necessity of inference.

Experimental integrity: The honest scoping is what I would single out. The reference semantics agreement is claimed for these controlled instances, and the caveat is stated — the benchmark shows the cost was avoidable here, not in general. Restricting to three efficiency-oriented models at low reasoning effort is a deliberate and stated choice, and the finding that more inference effort helps partially is reported rather than glossed. The limitation is that a single runtime anchors the ground truth, so generality across harness implementations is assumed rather than demonstrated.

Writing quality: The four question types are stated as tasks rather than as metrics, which makes the failure pattern readable — you can see why final-state prediction and cross-order reasoning are harder. Because the practical upshot is that an executable reference exists, a short section on when a reader should build one for their own harness would make the paper directly reusable rather than only informative.

Verdict: strong accept — it defines and measures a reasoning burden that agents impose on themselves, localises the failure to simulation rather than traversal, and shows with an independent exact method that the expensive path was optional.

Takeaways

• Separate traversal from simulation when diagnosing failures. Identifying affected components and predicting post-teardown state demand different operations, and only the second is hard.
• Build an executable reference when the answer is computable. If an exact procedure matches your runtime, the model’s inference effort is optional and its errors are avoidable.
• Sweep complexity rather than reporting one accuracy. Reliability that falls with the number of relevant interactions is a different finding from low accuracy overall.
• Price the remedy. Reporting that more inference effort helps, alongside what it costs per question, is what lets a reader decide whether to buy it.