Shadow Evaluations: Can AI Agents Actually Do Open-Ended AI Research?

Paper: 2607.27191 Authors: Peter Kirgis, Sayash Kapoor, Andrew Schwartz, Stephan Rabanser, David Africa, Konstantinos Voudouris, Viet Nguyen, Toby Pilditch, Magda Dubois, Harry Coppock Categories: cs.AI, cs.CY, cs.LG

The Gap

The field has a serious measurement problem when it comes to AI agents doing research. On one side, you have benchmarks like SWE-Bench or MLE-Bench that test agents on narrow, well-defined, verifiable tasks — fix this bug, build that model. These are clean and reproducible, but they completely exclude the open-ended, judgment-heavy core of real research. On the other side, you have people submitting AI-generated papers to blind peer review — which is overstretched, noisy, and plagued by inconsistent review quality. Neither tells us whether an agent can actually do research: formulate hypotheses, navigate dead ends, exercise judgment about what matters, and produce something a domain expert would recognize as a contribution.

This paper introduces a third path: shadow evaluations. Give the agent the same central research question that a human team tackled in a high-quality unpublished paper, let it run for days with real compute budgets, and then have the *original authors — the world’s leading experts on that exact question — grade the output. The logical chain is clean:

Problem: AI progress forecasts assume agents can automate research
    |
    v
But we have no good way to measure open-ended research ability
    |
    v
Existing evals are either too narrow (tasks) or too noisy (peer review)
    |
v
Shadow eval: same question, same resources, original authors grade
    |
    v
Result: agents can engineer, but cannot research
    |
    v
Conclusion: forecasts of imminent R&D automation are premature

The Increment

Before this paper, we either tested AI agents on toy engineering tasks or submitted their papers to unreliable peer review — both missing the point. After this paper, we have a repeatable evaluation methodology where domain experts directly judge whether an agent’s attempt at open-ended research passes muster, along with concrete evidence that current frontier agents fail at the research lifecycle despite excelling at engineering.

Core Mechanism

The shadow evaluation works as follows. The researchers identified two unpublished NeurIPS 2026 submissions — high-quality papers whose authors were willing to participate. For each paper, they extracted the core research question, any datasets or code the authors used, and relevant background context. They then gave a frontier AI agent (Claude 3.5 Sonnet with a custom scaffold) six days and thousands of dollars worth of compute to attempt to answer the same research question the human authors had tackled.

The agent was given access to tools: code execution, file editing, web search, and a structured workspace. It could run experiments, iterate on designs, and build whatever it needed. After the six-day run, the agents’ full logs, code, and any resulting outputs were sent to the original paper authors. These authors — who knew the problem better than anyone on earth — evaluated whether the agent’s work constituted meaningful progress toward answering the research question. A robustness check ran the same setup with a different model (GPT-4o) and scaffold to see if the failures were model-specific or structural.

[Unpublished NeurIPS 26 Paper]
    |
    +-- Extract: research question, data, context
    |
    v
[Agent + Scaffold + Tools]
    |
    +-- 6 days, $1000s compute
    +-- Code execution, experiments, iteration
    |
    v
[Agent Output: logs, code, results]
    |
    v
[Original Authors Evaluate]
    |
    +-- Did it answer the research question?
    +-- Does it constitute a contribution?
    |
    v
[Verdict: Rejected (both cases)]
    |
    v
[Failure Mode Analysis]
    |
    +-- 5 recurring patterns identified
    +-- Replicated with second model/scaffold

Structural metaphor: Think of it like a baking competition with a twist. Imagine you give a contestant the recipe name — “make a lemon tart that won the 2025 Paris pastry competition” — but not the recipe itself. You give them a fully stocked professional kitchen, six days, and unlimited ingredients. Afterward, the original pastry chef tastes the result and judges whether it’s competition-worthy.

The agent is the contestant: it can perfectly execute any technique (whisking, baking, plating) — the engineering is flawless. But it makes bad judgment calls: it picks the wrong variety of lemons, doesn’t adjust for altitude, over-complicates the crust, and when the first attempt tastes off, it doesn’t understand why well enough to course-correct. The original chef’s judgment is the key: they’re not checking whether the kitchen skills worked (they did), but whether the thinking behind the tart — the recipe design, the trade-offs, the creative pivots — was any good. It wasn’t. That’s the gap between engineering and research.

Key Concepts

• Shadow Evaluation: Imagine you’re a teacher. Instead of giving students a standardized test (narrow, verifiable) or asking a stranger to grade their essay (noisy, unreliable), you ask them to solve the same problem you yourself solved last year, and then *you grade it. You know exactly how hard the problem is, what pitfalls exist, and what a good answer looks like. That’s a shadow evaluation. The “shadow” part means the agent doesn’t replace the original researchers — it shadows them, attempting the same work in parallel. The power is that the grader has ground truth in their head.

• Open-ended vs. Verifiable Research Tasks: A verifiable task has a clear success condition — the code runs, the bug is fixed, the accuracy goes up. Open-ended research has no such binary signal. “Is this a meaningful contribution to the field?” is a judgment call, not a checksum. The paper argues (convincingly) that the gap between these two is exactly where current agents fall apart, because open-ended research requires taste, backtracking from dead ends, and knowing when “good enough” isn’t good enough.

• Instruction Drift: One of the five failure modes the authors identify. Over a long agent run, the agent gradually loses track of its original goal and constraints — like a game of telephone. It starts optimizing for something adjacent to the real objective. In one case, the agent spent enormous compute on engineering infrastructure that was tangential to the actual research question. Think of it as the agent’s equivalent of a PhD student who spends two years building a dataset nobody asked for instead of answering their thesis question.

Framework Shift

Before (mainstream approach):           After (this paper):

[Task Benchmark]                        [Shadow Evaluation]
 |                                        |
 +-- Narrow, verifiable tasks             +-- Open-ended research questions
 +-- Pass/fail metrics                    +-- Expert judgment
 +-- Reproducible but shallow             +-- Ecologically valid
 +-- Measures engineering                 +-- Measures research lifecycle
 |                                        |
 v                                        v
 "Agent can code"                  "Agent can engineer,
                                    but cannot research"

From narrow task completion to expert-judged research quality, the core shift is that we now have a way to measure whether AI agents can do the thinking part of research, not just the doing part.

Expert Assessment

Problem choice: This is a genuinely important gap. The AI safety and forecasting communities are making enormous bets on the assumption that AI will automate AI research soon. If that assumption is wrong, the downstream implications are massive. The authors are right that existing evaluations completely miss the mark for measuring open-ended research. This is not a manufactured problem — it’s one the field has been nervously avoiding because it’s hard to evaluate.

Method maturity: The shadow evaluation concept is clever and well-motivated. It’s not brute force — it’s a genuine methodological contribution. That said, the sample size is brutally small (two papers). The authors acknowledge this, but it limits the strength of any claims. There’s also a question of whether NeurIPS submissions are the right testbed — they’re a specific genre with specific expectations. A broader set of research tasks (theory, systems, applications) would strengthen the framework. The robustness check with a second model is a nice touch but doesn’t substitute for more cases.

Experimental integrity: The setup is honest. They gave the agents real resources, real time, and didn’t handicap them. The grading by original authors is a clever control — it eliminates the “who grades the grader” problem. The failure mode analysis is well-structured and specific. One concern: the agents are evaluated by people who are deeply invested in the value of human research, which could introduce bias, but this is hard to avoid and the authors are transparent about it.

Writing quality: The paper is clearly written and well-organized. The failure mode taxonomy is the strongest section — concrete, specific, and actionable. The weakest section is the related work, which feels perfunctory. A deeper engagement with the AI scientist literature (Sakana’s work, etc.) would strengthen the positioning. The discussion of limitations is honest but could go further on the question of whether better scaffolding or different agent architectures might change the picture.

Verdict: weak accept — The methodology is genuinely novel and the question is important, but two case studies are thin evidence for the weight of the claims. The framework earns its keep; the conclusions need more data.

Takeaways

Three things a practitioner can steal right now:

1. The shadow evaluation methodology itself. If you’re building AI agents for any domain where “quality” is a judgment call (not just pass/fail), you can replicate this: find an expert who recently solved a similar problem, give the agent the same question, let the expert grade. It’s more informative than any benchmark and cheaper than running a full peer review process.

2. The five failure mode taxonomy is a practical checklist for anyone building research agents. Poor judgment, uncreative pivots, ineffective backtracking, resource waste, and instruction drift — these are the things to watch for and scaffold against. If your agent is doing long-horizon work, you should build in checkpoints for each of these.

3. The engineering-research gap is real and specific. When someone tells you “AI agents can do AI research,” push back: they can do the engineering. The research lifecycle — scoping, judging, pivoting, knowing when to stop — is a different capability entirely. This distinction should inform how you design agent systems and what you promise stakeholders.