IdeaAMBIG: Finding the Gap in a Method Spec Is the Hard Part

Paper: 2609.10539 Authors: Yiling Ma, Yilun Zhao, Sihong Wu, Manasi Patwardhan, Arman Cohan Categories: cs.CL

The Gap

A research idea can be novel, coherent, and scientifically plausible — satisfying every criterion a reviewer applies — yet its proposed method may remain insufficiently specified for faithful implementation. Reviewing evaluates the idea; implementing requires the method, and those are different artifacts.

The paper names the missing property codification readiness: whether a specification provides sufficient methodological information for a competent implementer or coding agent to construct the intended method without unsupported assumptions. The clause doing the work is “without unsupported assumptions.” An underspecified method does not announce itself — it is implementable, in the sense that someone will implement something. The failure appears as a divergence between what was intended and what got built, discovered later.

That makes the property hard to evaluate informally, and it makes an evidence-grounded benchmark necessary rather than convenient.

   A RESEARCH IDEA CAN BE NOVEL, COHERENT, PLAUSIBLE --
   AND ITS METHOD STILL UNSPECIFIED

   a reviewer's criteria: NOVELTY, COHERENCE, PLAUSIBILITY
     -> the idea can satisfy ALL of them
        |
        v
   YET the proposed METHOD may remain INSUFFICIENTLY SPECIFIED
   FOR FAITHFUL IMPLEMENTATION
        <- reviewing evaluates the IDEA
        <- implementing requires the METHOD
        <- DIFFERENT ARTIFACTS

   [THE MISSING PROPERTY: CODIFICATION READINESS]
     whether a specification provides SUFFICIENT METHODOLOGICAL
     INFORMATION for a competent implementer OR CODING AGENT to
     construct the intended method WITHOUT UNSUPPORTED ASSUMPTIONS
        <- "WITHOUT UNSUPPORTED ASSUMPTIONS" is the clause doing the work
        <- an underspecified method DOES NOT ANNOUNCE ITSELF:
           it is implementable, in the sense that someone will
           implement SOMETHING
        -> the failure appears as DIVERGENCE between what was INTENDED
           and what got BUILT, discovered LATER
        |
        v
   -> hard to evaluate INFORMALLY
   -> an EVIDENCE-GROUNDED benchmark becomes NECESSARY, not merely
      convenient

The Increment

One sentence: Before this paper, whether a method specification was implementable was a matter of judgement; after it, a 660-instance benchmark with real and injected defects separates three capabilities and shows localisation is where models fail.

Core Mechanism

The benchmark’s two halves are sourced differently, and that is the design. 163 real-world gaps come from reproducibility reports and GitHub issues — defects that actually impeded someone — and 497 controlled synthetic gaps are injected into codification-ready references. The real instances supply ecological validity: these are the gaps that mattered. The synthetic ones supply control: because a codification-ready reference is the starting point, the injected defect is the only reason an instance is unready, so a model’s failure is attributable. Neither half alone would support the claims; together they let the paper say something about real gaps and something causal about detection.

Three capabilities are evaluated, and their separation is what turns a score into a diagnosis:

• Codification-readiness assessment — is this specification ready?
• Defect localization — where is the gap?
• Clarification action generation — what would resolve it?

The evaluation conditions are deliberately unequal, and that is the sharpest design choice: defect localization receives only the specification, whereas clarification additionally receives the annotated defect. So the two tasks are not parallel; clarification is measured with the answer already supplied. That asymmetry is what makes the results interpretable, because it isolates the hard sub-problem from the easy one.

And the results show exactly that separation. Across 13 LLMs, the best model achieves 9.6% Macro Defect Recovery Rate on real-world instances — that is with the specification alone — but 80.6% Macro Clarification Action Success Rate when given the defect. Read together, these two numbers say something specific: models are good at resolving a gap once told where it is, and poor at finding it. The comparison is only meaningful because the conditions were set up to isolate that.

An oracle study bounds the value: supplying the gold resolution raises the downstream codification-ready rate from 14% to 98%. So the gap between 14% and 98% is the space that better clarification could recover — while the 9.6% localisation figure says what currently stands in the way of getting there. The oracle also serves as a validity check: it confirms the resolutions are sufficient, so the benchmark is measuring sufficiency rather than something orthogonal to it.

The summary is stated plainly: across all evaluated models, defect localization is the main bottleneck, with stronger clarification given the defect. That is a conclusion about where research effort should go, and it is the opposite of where a benchmark focused on generation would point.

   THE BENCHMARK'S TWO HALVES ARE SOURCED DIFFERENTLY -- THAT IS THE DESIGN
     163 REAL-WORLD gaps from REPRODUCIBILITY REPORTS and GITHUB ISSUES
       <- defects that ACTUALLY IMPEDED someone
       -> ECOLOGICAL VALIDITY: these are the gaps that MATTERED
     497 CONTROLLED SYNTHETIC gaps INJECTED INTO CODIFICATION-READY
       REFERENCES
       <- the reference is READY, so the injected defect is the ONLY
          reason an instance is unready
       -> a model's failure is ATTRIBUTABLE
     <- neither half alone would support the claims; together they let
        the paper say something about REAL gaps and something CAUSAL
        about DETECTION

   THREE CAPABILITIES ARE EVALUATED -- their separation turns a score
   into a DIAGNOSIS
     CODIFICATION-READINESS ASSESSMENT -- is this specification ready?
     DEFECT LOCALIZATION            -- WHERE is the gap?
     CLARIFICATION ACTION GENERATION -- what would RESOLVE it?

   THE EVALUATION CONDITIONS ARE DELIBERATELY UNEQUAL -- the sharpest
   design choice
     DEFECT LOCALIZATION receives ONLY THE SPECIFICATION
     CLARIFICATION ADDITIONALLY RECEIVES THE ANNOTATED DEFECT
       <- the two tasks are NOT PARALLEL: clarification is measured
          WITH THE ANSWER ALREADY SUPPLIED
       -> that asymmetry is what ISOLATES THE HARD SUB-PROBLEM from
          the EASY one

   RESULTS -- and they show EXACTLY THAT SEPARATION
     across 13 LLMs, the BEST model achieves
       9.6%  MACRO DEFECT RECOVERY RATE on REAL-WORLD instances
              <- that is WITH THE SPECIFICATION ALONE
       80.6% MACRO CLARIFICATION ACTION SUCCESS RATE given the defect
     READ TOGETHER: models are GOOD AT RESOLVING A GAP ONCE TOLD
       WHERE IT IS, and POOR AT FINDING IT
       <- the comparison is meaningful ONLY because the conditions were
          set up to ISOLATE that

   AN ORACLE STUDY BOUNDS THE VALUE
     supplying the GOLD RESOLUTION raises the DOWNSTREAM
     CODIFICATION-READY RATE from 14% to 98%
       -> the span from 14% to 98% is the space BETTER CLARIFICATION
          could recover
       -> while 9.6% LOCALISATION says what currently STANDS IN THE WAY
       <- the oracle is also a VALIDITY CHECK: it confirms the
          resolutions ARE SUFFICIENT
          -> the benchmark measures SUFFICIENCY, not something
             orthogonal to it

   SUMMARY, STATED PLAINLY
     across all evaluated models, DEFECT LOCALIZATION IS THE MAIN
     BOTTLENECK, with stronger clarification given the defect
       <- a conclusion about WHERE RESEARCH EFFORT SHOULD GO
       <- and the OPPOSITE of where a benchmark focused on GENERATION
          would point

Think of it as the difference between fixing a bridge and noticing which plank is rotten. Given an inspection report that names the rotten plank, replacement is a solved kind of task — there is a spec, and the work is bounded. Noticing it is the whole job, and it is the part nobody can do by being handed the conclusion. The paper’s conditions make this concrete: the same models score 80.6% at the fixing task when told the location, and 9.6% at finding it. And the oracle figure — 14% to 98% — is the equivalent of showing that a fully correct inspection leads to a sound bridge, so the low number really is about inspection rather than about the repairs being wrong.

Key Concepts

• Codification readiness: whether a specification suffices for a competent implementer or agent to build the intended method without unsupported assumptions. It is the property that review criteria do not capture.
• Two sourcing strategies in one benchmark: real gaps from reproducibility reports and issues, plus synthetic gaps injected into ready references. Validity and causality, respectively.
• Unequal evaluation conditions by design: localisation gets the specification alone, clarification gets the defect. The asymmetry is what makes the comparison isolate the hard sub-problem.
• The 9.6% versus 80.6% split: findings-are-hard, fixes-are-easy. It is the empirical form of the claim that localisation is the bottleneck.
• Oracle-bounded headroom: 14% to 98% downstream readiness with the gold resolution. It measures the recoverable space and confirms the resolutions are sufficient.

Framework Shift

Before (readiness as a matter of judgement):
  review the idea for novelty, coherence, plausibility
  -> implementation-divergence discovered later
  -> no way to measure whether a method is specified enough
  -> no account of which sub-problem is hard

After (readiness measured, capabilities separated):
  660 instances: 163 real gaps + 497 injected into ready references
  localisation gets the spec; clarification gets the defect too
  -> 9.6% localisation on real-world instances
  -> 80.6% clarification success given the defect
  -> gold resolution lifts readiness from 14% to 98%
  -> localisation is the bottleneck

From judging whether an idea is good, to measuring whether its method is specified enough to build, the core shift is that the failure to implement an idea faithfully is a detectable, localisable defect — and models can fix such defects far better than they can find them.

Expert Assessment

Problem choice: Excellent, and it identifies the gap between how research is evaluated and what implementation requires. Peer review assesses ideas; implementation consumes methods; and the divergence between them is a real and under-measured cost, made more pressing by coding agents that will happily build the wrong thing from an ambiguous spec.

Method maturity: The design is strong in two specific ways. The dual sourcing gives both ecological validity and causal attribution, which is hard to achieve in one benchmark. And the deliberately unequal evaluation conditions — localisation without the defect, clarification with it — are what make the headline split interpretable rather than merely reported. The oracle study does double duty, bounding the recoverable headroom and validating that the resolutions are actually sufficient.

Experimental integrity: Reporting that 9.6% is achieved by the best of thirteen models on real instances is candid about difficulty, and it is stated in the right place rather than buried. The separation of three capabilities means a reader can see which one failed, and the finding that clarification is stronger given the defect is a consequence of the design rather than a post hoc comparison. The main limitation is the real-world half’s size: 163 instances sourced from reports and issues is enough to show the capability is poorly served by current models, but small for characterising how the defect distribution looks across fields.

Writing quality: The three capabilities are named with one-line definitions, which makes the results table legible before the details. Because the practical implication is “build better defect detectors, not better clarifiers”, a short passage on what distinguishes a localisable defect from an unlocalisable one — is it always a missing element, or sometimes an inconsistency between elements? — would give the finding a direction.

Verdict: strong accept — it names and measures a property that peer review does not assess, designs conditions that isolate localisation from clarification, and reports that the hard part is finding the defect rather than resolving it.

Takeaways

• Separate finding a defect from fixing it. Conditions that hand over the defect measure repair; only conditions without it measure detection.
• Source a benchmark two ways. Real instances give validity, injected defects give attribution, and the claims need both.
• Use an oracle to bound headroom. Reporting 14% to 98% with the gold resolution says how much is recoverable and confirms the resolutions suffice.
• Ask what a low score is actually blocked on. Here localisation blocks clarification, which points effort at detection rather than generation.