Can AI Coding Agents Actually Do Science? Testing LLMs on Materials Science Reproducibility

Paper: 2605.00803 Authors: Ziyang Huang, Yi Cao, Ali K. Shargh, Jing Luo, Ruidong Mei, Mohd Zaki, Zhan Liu, Wyatt Bunstine, William Jurayj, Somdatta Goswami Categories: cs.SE, cs.AI, cs.CL

The Gap

LLM-based coding agents crush software engineering benchmarks like HumanEval and SWE-bench. But those benchmarks test isolated coding tasks with clear specs and standard toolchains. Scientific computing is different: you’re reconstructing underspecified procedures from paper text, navigating domain-specific tools you’ve never seen, and judging whether your results actually support a scientific claim. No one has tested whether coding agents can handle this full pipeline.

The gap: we know agents can write code, but we don’t know if they can do computational science end-to-end. Prior benchmarks measure coding ability in isolation. This paper asks whether that ability transfers to reproducing scientific findings, where the real challenge is navigating ambiguity, specialized tools, and scientific reasoning.

Problem: Coding benchmarks != scientific workflows
   |
   v
Assumption: Scientific reproducibility requires
            procedure recovery + tool navigation + claim validation
   |
   v
Method: AutoMat benchmark with real materials science claims
        + multi-stage evaluation (procedure, execution, validation)
   |
   v
Evidence: Best agent achieves 54.1% success
          Failure modes: incomplete procedures (38%), 
                        methodological deviations (22%),
                        execution fragility (18%)
   |
   v
Conclusion: Current agents struggle with scientific workflows
            Gap widens when procedures must be inferred from text

The Increment

One sentence: Before this paper, we knew LLMs could code; after it, we know they can’t reliably reproduce computational science findings, failing most when procedures are underspecified.

Core Mechanism

AutoMat is a three-stage gauntlet. First, the agent reads a materials science claim from a paper and must recover the computational procedure—often underspecified or scattered across methods sections. Second, it executes that procedure using specialized tools like VASP, Quantum ESPRESSO, or ASE, which have their own quirks and conventions. Third, it interprets the results and determines whether they support or contradict the original claim.

The benchmark curates claims from real papers, working with domain experts to establish ground truth. Each claim comes with varying levels of procedural detail: some include full code, others just describe the approach in prose. The evaluation tracks success at each stage—did the agent recover the right procedure? Did it execute without errors? Did it reach the correct scientific conclusion?

Paper claim --> [Agent] --> Procedure recovery
                              |
                              v
                         Tool execution
                         (VASP, QE, ASE...)
                              |
                              v
                         Result interpretation
                              |
                              v
                         Claim validation
                         (support/contradict)

Think of AutoMat as a cooking competition where the recipe is written in a foreign language with half the steps missing. The agent must figure out what ingredients to use (procedure recovery), operate unfamiliar kitchen equipment (tool navigation), taste the result, and decide if it matches the dish described in a food magazine (claim validation). A coding benchmark is like asking someone to chop an onion—useful, but it doesn’t tell you if they can cook a meal. AutoMat tests the full meal.

Key Concepts

• Procedure recovery: Scientific papers rarely include executable code. Methods sections describe approaches in prose, often omitting parameters, convergence criteria, or preprocessing steps that experts consider “obvious.” Procedure recovery means reconstructing a runnable workflow from this incomplete description. It’s like being told “simmer until done” without a temperature or time—you need domain knowledge to fill the gaps.

• Claim validation: Unlike software tests with binary pass/fail, scientific claims involve judgment. If a paper claims “material X is stable,” you must run simulations, extract energy values, compare to baselines, and decide if the evidence supports that conclusion. The agent needs to understand what “stable” means in context, what threshold matters, and whether small numerical differences are significant or noise.

• Execution fragility: Scientific tools are brittle. VASP expects specific file formats. Quantum ESPRESSO has cryptic error messages. A missing pseudopotential or wrong unit cell parameter crashes the whole pipeline. Unlike web dev where you can Google the error, scientific computing errors require domain expertise to diagnose. Agents must handle this fragility without human intervention.

Framework Shift

Before (coding benchmarks):          After (AutoMat):

Spec --> Code --> Test               Paper text --> Procedure recovery
  |        |        |                     |              |
Clear   Standard  Binary            Ambiguous      Domain-specific
input   tools     output            description    tools (VASP, QE)
                                          |              |
                                          v              v
                                    Execution --> Interpretation
                                       |              |
                                    Fragile       Requires scientific
                                    toolchain     judgment

From isolated coding tasks to end-to-end scientific workflows, the core shift is from executing clear specs with standard tools to reconstructing ambiguous procedures with specialized tools and validating results against scientific claims.

Expert Assessment

Problem choice: This is a real gap. The field has been celebrating agent performance on SWE-bench without asking whether that success transfers to domains where specs are ambiguous and tools are specialized. Scientific reproducibility is a high-stakes application where we need to know the limits. The problem sits at the intersection of AI-for-science and agent evaluation—timely and important.

Method maturity: The benchmark design is solid. Working with domain experts to curate claims and establish ground truth is the right move. The three-stage evaluation cleanly separates failure modes. However, the paper doesn’t explore whether agents could improve with domain-specific prompting, retrieval-augmented generation over tool docs, or iterative refinement. It’s a diagnostic benchmark, not a methods paper, but some ablations on agent design would strengthen it.

Experimental integrity: Baselines are fair—they test multiple models (GPT-4, Claude, Gemini) and agent frameworks (ReAct, reflexion, multi-agent). The 54.1% success rate is sobering but believable given the task difficulty. Error analysis is thorough, breaking down failures by stage and root cause. One concern: the benchmark is small (exact size not specified in abstract), which limits statistical power. Also, no comparison to human performance—how often do grad students successfully reproduce these claims?

Writing quality: The abstract is clear and well-structured. The framing is strong—positioning this as both a benchmark and a diagnostic tool. If I were reviewing, I’d push the authors to expand the related work section to better situate this against other scientific benchmarks (e.g., ScienceQA, PubMedQA) and to include human baselines. The error analysis section is likely the paper’s strongest contribution and should be front-loaded.

Verdict: weak accept — Addresses a real gap with a well-designed benchmark, but limited scale and lack of human baselines weaken the empirical contribution. The error taxonomy is valuable for the community.

Takeaways

If you’re building agents for specialized domains, steal the three-stage evaluation framework: procedure recovery, execution, validation. It cleanly separates failure modes and tells you where your agent breaks. Also, the finding that agents perform worst when procedures must be inferred from text is a warning: if your domain has underspecified workflows, current agents will struggle. Consider building retrieval systems over domain-specific documentation or few-shot examples of complete procedures. Finally, the execution fragility issue suggests that agents need better error recovery—not just retry logic, but the ability to diagnose domain-specific errors and adjust parameters. That’s a concrete research direction.