Measuring Mid-2025 LLM-Assistance on Novice Performance in Biology

Paper: 2602.16703 Authors: Shen Zhou Hong, Alex Kleinman, Alyssa Mathiowetz, Adam Howes, Julian Cohen, Suveer Ganta, Alex Letizia, Dora Liao Categories: cs.CY, cs.AI

The Gap

Large language models score impressively on biological benchmarks—protein folding, sequence analysis, protocol design. This has raised biosecurity concerns: if LLMs can ace these tests, might they enable novice actors to acquire dangerous laboratory skills?

But there’s a critical question nobody had rigorously answered: Do benchmark scores translate to real-world laboratory performance? Benchmarks are in silico—computational tasks, text-based questions, simulated scenarios. Real biology happens in physical labs with pipettes, cell cultures, and equipment that doesn’t always behave as expected.

The gap between benchmark performance and practical utility is unknown. We don’t know if a model that scores 90% on a virology quiz actually helps a novice successfully culture cells or perform reverse genetics. Without this validation, biosecurity assessments based on benchmarks alone are speculative.

Known:
  LLMs perform well on biological benchmarks
  [In silico, text-based, controlled conditions]

Unknown:
  Do LLMs improve novice performance in physical labs?
  [Wet lab, hands-on, real-world constraints]

Gap: Benchmark scores → Real-world utility
     No empirical validation
     Biosecurity assessments lack grounding

The Increment

One sentence: Before this study, we had no rigorous evidence that LLM benchmark performance translates to improved novice laboratory skills; after this RCT with 153 participants, we know mid-2025 LLMs provide modest benefits (1.4× success rate) but don’t substantially increase workflow completion, revealing a benchmark-reality gap.

Core Mechanism

The researchers conducted a pre-registered, investigator-blinded, randomized controlled trial from June-August 2025 with 153 participants. Participants were novices (no prior molecular biology experience) randomly assigned to either LLM assistance or internet-only control.

The tasks modeled a viral reverse genetics workflow—the process of creating viruses from genetic sequences. This workflow has dual-use implications (legitimate research vs. potential misuse). Five tasks were tested:

1. Protocol design: Plan experimental procedures
2. Primer design: Design DNA primers for amplification
3. Cell culture: Grow and maintain cells
4. Transfection: Introduce genetic material into cells
5. Virus rescue: Recover infectious virus from transfected cells

The primary endpoint was workflow completion—successfully completing all five tasks. Secondary endpoints included individual task success rates and progression through intermediate steps.

RCT Design:
  n = 153 novices (no prior molecular biology experience)
  
  Random assignment:
    LLM arm: Access to GPT-4 and Claude
    Control arm: Internet search only
  
  Tasks: Viral reverse genetics workflow
    1. Protocol design
    2. Primer design
    3. **Cell culture**
    4. Transfection
    5. Virus rescue
  
  Measurement:
    Primary: Complete workflow (all 5 tasks)
    Secondary: Individual task success
                Intermediate step progression

Think of it like comparing GPS navigation (LLM) versus paper maps (internet) for novice drivers attempting a complex route. The GPS might help with individual turns, but completing the entire journey depends on many factors beyond navigation—traffic, vehicle handling, decision-making under uncertainty.

Key Concepts

• Randomized controlled trial (RCT): The gold standard for causal inference. Participants are randomly assigned to treatment (LLM) or control (internet), eliminating selection bias. Pre-registration prevents p-hacking. Investigator blinding reduces experimenter bias. This design lets us confidently attribute differences to LLM assistance rather than confounding factors.

• Workflow completion vs. task success: Workflow completion requires succeeding at all five sequential tasks—a high bar that tests whether LLMs enable end-to-end capability. Task success measures individual components. The distinction matters: a novice might successfully culture cells (one task) but fail to complete the full workflow due to downstream failures.

• Benchmark-reality gap: The difference between performance on standardized tests (benchmarks) and performance in real-world applications. Benchmarks control for confounds and measure specific capabilities in isolation. Reality is messy—tacit knowledge, equipment variability, time pressure, incomplete information. A model can score perfectly on a protocol design benchmark but provide advice that fails in practice due to unstated assumptions or context-dependent factors.

Framework Shift

Before (benchmark-based assessment):    After (RCT validation):

Evaluation: In silico benchmarks        Evaluation: Physical lab tasks
            Text-based questions                   Hands-on procedures
            Controlled conditions                  Real-world constraints

Assumption: Benchmark → Real utility    Finding: Modest benefit (1.4×)
            [Unvalidated]                         No workflow completion boost
                                                  [Empirically grounded]

Biosecurity: High benchmark scores      Biosecurity: Benchmark-reality gap
             = High risk?                          Risk assessment needs
             [Speculative]                         physical validation
                                                  [Evidence-based]

One sentence: From speculative biosecurity assessments based on in silico benchmarks to empirically grounded understanding that mid-2025 LLMs provide modest task-level benefits without substantially enabling end-to-end workflows.

Expert Assessment

Problem choice: This is exactly the right question. The field has been making biosecurity assessments based on benchmark performance without validating whether those benchmarks predict real-world capability. The RCT design is appropriate and the viral reverse genetics workflow is a reasonable proxy for dual-use skills. Well-motivated and timely.

Method maturity: The experimental design is rigorous—pre-registered, blinded, randomized, adequate sample size (n=153). The choice of tasks is thoughtful, covering the full workflow rather than isolated steps. The Bayesian modeling adds nuance beyond frequentist null hypothesis testing. The ordinal regression for intermediate steps is a smart way to capture partial progress.

Experimental integrity: The results are credible and honestly reported. No workflow completion difference (5.2% vs 6.6%, P=0.759) is the headline, but they don’t bury the secondary findings—cell culture showed a trend (68.8% vs 55.3%, P=0.059) and ordinal regression suggests LLMs help with intermediate steps. The 1.4× effect size (95% CrI 0.74-2.62) is modest and the confidence interval crosses 1, indicating uncertainty. This is honest science.

Writing quality: Clear exposition of motivation, methods, and results. The discussion appropriately emphasizes the benchmark-reality gap and the need for physical validation. Could benefit from more speculation about why the effect is modest—is it tacit knowledge? Equipment handling? Time pressure? The limitations section is appropriately cautious.

Verdict: strong accept — Rigorous empirical validation of a critical assumption in AI biosecurity, revealing that benchmark performance doesn’t straightforwardly translate to real-world capability, with important implications for risk assessment.

Takeaways

• Validate in the real world: If you’re assessing AI capabilities for high-stakes applications (biosecurity, medical diagnosis, autonomous systems), don’t rely solely on benchmarks. Run physical-world validation studies. Benchmarks are useful for tracking progress but poor predictors of practical utility.

• Modest effects matter: The 1.4× effect size seems small, but in biosecurity contexts, even modest capability increases could matter if they lower barriers for malicious actors. The finding that LLMs help with intermediate steps (81-96% posterior probability) suggests they provide scaffolding even when not enabling full workflows.

• Task vs. workflow distinction: When evaluating AI assistance, distinguish between task-level success and end-to-end workflow completion. A system might help with individual steps without enabling the full process. This matters for both capability assessment and interface design.

• Temporal dynamics: This is a snapshot of mid-2025 capabilities. The authors correctly note that “model capabilities and user proficiency evolve.” Continuous monitoring is needed—today’s modest effect could become substantial as models improve or as users learn to leverage them better.

• Methodological template: The RCT design, pre-registration, blinding, and Bayesian analysis provide a template for evaluating AI assistance in other domains. This approach is transferable beyond biosecurity to any field where benchmark-reality gaps are suspected.