ABC-Bench: The First Benchmark for AI Agents Doing Real Biology Lab Work

Paper: 2606.11150 Authors: Andrew Bo Liu, Samira Nedungai, Bryce Cai, Alex Kleinman, Harmon Bhasin, Seth Donoughe Categories: cs.AI, cs.CY

The Gap

Most biosecurity evaluations for LLMs ask factoid questions: “What toxin causes botulism?” or “List genes needed for X.” They test static knowledge. Meanwhile, LLM agents – models that can write code, call APIs, and control hardware – are beginning to perform tasks that previously required a human with a bench and a pipette. No existing benchmark measures whether an AI agent can actually do wet-lab biology: operate a liquid-handling robot, design a DNA fragment for assembly, or evade a screening filter. This paper builds that missing bridge.

Existing prior art includes:

• CTI (BioSafety benchmarks): multiple‑choice questions about dangerous pathogens.
• Headroom (BioCapabilities eval): yes/no on whether an LLM can answer biology questions above expert level.
• Singh et al. (Dual‑Use eval): static list of “dangerous capabilities” with human‑rated scores.

None involve acting in the world – executing code that moves liquid, or generating sequences that pass screening. The gap is a benchmark that demands agentic execution, not just recall.

[AI bio risk exists] -> [Current evals: static Q&A only]
                                    |
                                    v
    [We need to test: can agents DO bio lab tasks?]
        |
        +--------+--------+
        |        |        |
        v        v        v
  [Task 1]  [Task 2]  [Task 3]
  Robot op   DNA des  Screen ev
        |        |        |
        +--------+--------+
        |
        v
[LLM agents > human median]
        |
        v
[Wet-lab validated for Task 1]
        |
        v
[Conclusion: track agentic bio capabilities]

The Increment

One sentence: Before ABC-Bench, we only knew what AI *knows about biology; after ABC-Bench, we know what AI can do in a biology lab – and the answer is “more than the median human.”

Core Mechanism

ABC-Bench isn’t a single test – it’s a three‑station exam designed to cover distinct agentic competencies relevant to biosecurity.

Station 1 – Liquid Handling Robot (OpenTrons): The agent receives a description of a molecular biology protocol (e.g., “assemble a plasmid from two fragments using Gibson assembly”). It must write Python code for the OpenTrons robot to execute. Success is measured by whether the code runs without error and, in wet‑lab validation, produces the correct DNA construct.

Station 2 – DNA Fragment Design for in vitro assembly: The agent is given a target gene sequence and a list of available DNA parts. It must design several overlapping DNA fragments that, when assembled via methods like Gibson or Golden Gate, reconstitute the target. Evaluation uses a simulator that checks fragment overlap, melting temperatures, and assembly feasibility.

Station 3 – DNA Synthesis Screening Evasion: The agent is given a DNA sequence encoding a known toxin (e.g., botulinum neurotoxin). It must obfuscate the sequence (e.g., codon‑modify, rearrange gene order, add filler) so that it passes a hypothetical screening filter based on BLAST against a watch‑list. Success means the modified sequence is not flagged.

Data flow: each task starts with a task prompt (text description + optional files) fed to the LLM agent. The agent outputs a deliverable (code, DNA sequences, etc.). That deliverable is then evaluated automatically (simulator) or manually (wet‑lab). For the robot task, the code is also physically run in a wet lab to confirm real‑world performance.

[Task Prompt]                        [Evaluation]
      |                                   ^
      v                                   |
[LLM Agent] -> [Code / Sequence] --------> [Simulator or Wet-lab]
                     |
                     v
              [Success Metric]
                    |
                    +-------- (binary: pass/fail, or continuous score)

Structural metaphor: The Lab Intern Exam

Imagine a university lab hiring a summer intern. To test their readiness, the lab director sets up three stations:

1. Robot Chef Station – The intern must program a kitchen robot to follow a complex cooking recipe (e.g., make a layered cake with frosting). The director checks if the robot’s movements produce the correct cake, and if the intern’s code handles edge cases (e.g., missing ingredient).
   Mapping: Robot = OpenTrons; Recipe = molecular biology protocol; Intern’s code = Python script; Cake = assembled DNA.

2. Recipe Designer Station – Given a list of pantry ingredients and a picture of the final dish, the intern must write down a recipe with exact measurements and steps. The director simulates the recipe to see if the dish comes out right.
   Mapping: Pantry = available DNA parts; Dish = target gene; Recipe = DNA fragment design; Simulation = assembly checker.

3. Customs Inspector Station – The intern is handed a banned ingredient (e.g., a dangerous chemical) and told to smuggle it through airport security. They must disguise it – change its label, mix with benign compounds, repackage – so that X‑ray and chemical tests don’t flag it.
   Mapping: Banned ingredient = toxin gene; Disguise = sequence obfuscation (codon changes, shuffling); Security test = BLAST screening.

The director (ABC-Bench) then compares the intern’s performance to that of a “median working biologist” (human baseline). The key insight: previous exams only asked interns about cooking theory; ABC‑Bench makes them actually cook.

Key Concepts

• Agentic Capability: The ability of an LLM not just to produce text, but to exert causal influence on the external world – e.g., by generating executable code that moves robot arms, or by producing DNA sequences that, if physically synthesized, would create a real biological agent. This is distinct from “knowledge” because it requires understanding of context, error‑handling, and low‑level details like pipetting volumes and timing.

  Example: An LLM might know that Gibson assembly uses a 50°C incubation. But to write code for the robot, it must also know that the robot’s pipette tip can only aspirate 200 µL at a time, and that the incubation step must be preceded by a “bead‑break” command. That extra layer of execution is agentic.

• Dual‑Use Biology Task: A task that has legitimate scientific use but could also be misused to cause harm. Designing DNA fragments is essential for synthetic biology, but the same skills could be used to assemble a pathogen genome. The key is that the capability itself is not “good” or “bad” – the context and intent decide.

  Example: The screening‑evasion task is explicitly dual‑use: the same obfuscation techniques that help a company protect trade secrets could also be used to smuggle a dangerous gene past regulatory filters.

• Wet‑Lab Validation: Running the agent‑generated code on physical laboratory equipment with real reagents, then analyzing the output (e.g., DNA sequencing) to confirm that the intended biological construct was actually created. This is the gold standard of evaluation – a simulation might miss real‑world issues like evaporation, tip clogging, or temperature gradients.

  Example: For the OpenTrons task, the authors ran the LLM’s Python script on an actual robot in a benchtop incubator, extracted the DNA from the reaction, and sequenced it. The sequencing results matched the expected DNA sequence, proving the code worked outside a simulator.

Framework Shift

Compare the old mainstream approach (static Q&A benchmarks) with ABC‑Bench’s approach (agentic execution).

Before (mainstream approach):           After (this paper):
[Q: "What is the optimal temp for]     [Task: "Write code to assemble two]
[ Gibson assembly?" ]                   [ fragments on a robot at 50degC"]
      |                                        |
      v                                        v
[LLM outputs: "50°C"]                    [LLM outputs Python script]
      |                                        |
      v                                        v
[Compare to human answer]                 [Execute robot + check DNA seq]
      |                                        |
      v                                        v
[Binary correct/incorrect]               [Continuous success + wet-lab]

From passive knowledge retrieval to active tool‑mediated execution, the core shift is framing capability assessment around what AI can *do in the world rather than what it can say.

Expert Assessment

Problem choice: Real gap, not manufactured. Biosecurity risk from AI is a hot policy topic, and existing evaluations are embarrassingly shallow. This paper targets the most dangerous part – when AI can *execute biology tasks autonomously. The timing is perfect: as of 2025, agents like GPT‑4 with code interpreter and tool use are already being used in academic labs.

Method maturity: Clever but not groundbreaking. The tasks are well chosen (cover three threat vectors), but the benchmark construction is straightforward – “take standard lab protocols, turn them into prompts, measure success.” There is no novel algorithmic contribution. The strength is in the careful human baseline collection and wet‑lab validation. Simpler alternatives exist (e.g., “just ask an expert to judge the code”) but they are less reproducible.

Experimental integrity: Mostly solid, but a few red flags. (1) The human baseline uses *median performance, not expert. That means a few very poor humans drag down the median; the top human expert likely far exceeds all tested agents. The authors acknowledge this but the headline “outperformed median human” is a bit cheap. (2) Wet‑lab validation only done on one task (robot) and only with one model (o4‑mini‑high). We don’t know if the other tasks’ code would work in reality. (3) The screening evasion task uses a simple BLAST filter; real screening systems are more sophisticated (AI‑based, homology not sequence identity). Still, it’s a start.

Writing quality: Clear and well‑organized. The authors did not waste space. The biggest weakness is the methodology section – it’s dense and could benefit from a worked example of each task (a full trace of input → agent output → evaluation). That section, if rewritten with a concrete example walkthrough, would vault the paper to top‑tier clarity.

Verdict: Weak accept – A valuable benchmark that fills a genuine gap, but lacks novelty in method and the human baseline comparison is slightly tilted. Useful for the biosecurity community; essential for policymakers.

Takeaways

• Task template for agentic evaluation: The three‑task structure (robot control, design, evasion) can be ported to other domains (chemistry, manufacturing, cyber). The key is to pick tasks that require external action not just text generation, and to include a human‑in‑the‑loop wet‑lab validation step for at least one task.

• Human baseline collection matters: The authors recruited 30 biology experts and had them perform the tasks *in silico (no wet lab for humans either). That is expensive but essential. For any agentic benchmark, you must know what “good enough” looks like.

• Surprise result: All tested LLM agents *outperformed the median human. That should be a wake‑up call. Even if the top human is still far ahead, the agents are already more reliable than the average bench scientist on these specific tasks. That has immediate implications for lab safety training and code review policies.