Benchmarking LLM Agents on Meta-Analysis: The Screening Bottleneck

Paper: 2606.17041 Authors: Anzhe Xie, Weihang Su, Yujia Zhou, Yiqun Liu, Qingyao Ai Categories: cs.CL, cs.IR

The Gap

Meta-analysis is the gold standard for evidence synthesis but requires a multi-step pipeline: (1) retrieve candidate studies, (2) screen them against PI/ECO criteria (Population, Intervention, Comparison, Outcome), and (3) aggregate statistically. Existing benchmarks like BioASQ or FEVER focus on single-step question answering, not this structured workflow. They lack ground-truth annotations for the full pipeline — in particular, they don’t provide verified included studies, hard negatives that are topically similar but PI/ECO-ineligible, nor the complete search strategies. This means we can’t pinpoint where LLM agents fail: is it retrieval, screening, or synthesis? This paper fills that gap with MetaSyn.

Problem: Need to evaluate full meta-analysis pipeline
    |
    v
Assumption: Existing benchmarks are not sufficient
    |
    v
Method: MetaSyn dataset + 12 pipeline configurations
    |
    v
Evidence: 90.9% recall ceiling at K=200, but max 52.7% on included studies
    |
    v
Conclusion: Screening is the critical bottleneck

The Increment

One sentence: Before MetaSyn, no dataset existed to evaluate the full retrieval-screening-synthesis pipeline of meta-analysis; after MetaSyn, we know that current LLM agents fail catastrophically at the screening step.

Core Mechanism

MetaSyn contains 442 expert-curated meta-analyses from Nature Portfolio journals. For each entry, the authors provide:

• A research question and its PI/ECO criteria
• A retrieval corpus of 140k PubMed articles (official search results from the original meta-analysis)
• Verified positive studies (the papers the experts actually included)
• Hard negatives (topically similar articles that pass retrieval but fail PI/ECO)
• The complete search strategy and date bounds

The paper then benchmarks 12 pipeline configurations: 9 RAG variants (different chunking, embedding, retrieval methods) and one protocol-driven agent (a system that follows the official search strategy step-by-step). They use stage-attributed metrics to measure recall at each step: retrieval recall (fraction of positive studies in the top K), screening precision (fraction of positive studies among screened candidates), and end-to-end recall (fraction of positive studies eventually included).

[Method internals: MetaSyn dataset structure and evaluation flow]

+----------------------------------+
|  MetaSyn Dataset (442 entries)   |
|  +---------------+ +----------+  |
|  | Query +       | | Positive |  |
|  | PI/ECO        | | Studies  |  |
|  +---------------+ +----------+  |
|  +---------------+ +----------+  |
|  | Hard Negatives| | Search   |  |
|  | (topical but  | | Strategy |  |
|  |  ineligible)  | | + bounds |  |
|  +---------------+ +----------+  |
+-----------+----------------------+
            |
            v
+--------------------------+
| Pipeline Configuration   |
| (RAG or protocol-driven) |
| -> retrieval             |
| -> screening             |
| -> synthesis (optional)  |
+--------------------------+
            |
            v
+--------------------------+
| Stage-attributed metrics |
| Retrieval recall @K      |
| Screening precision      |
| End-to-end recall        |
+--------------------------+

Let’s use the analogy of a detective solving a cold case. The query + PI/ECO criteria are like the detective’s case file: the victim’s description, the modus operandi, the evidence needed. The retrieval corpus is the city’s entire arrest database — 140k records. The positive studies are the true suspects who committed the crime. The hard negatives are red herrings: people with similar rap sheets but airtight alibis (they fail PI/ECO). The search strategy is the detective’s playbook of which precincts to check, what dates to focus on. The protocol-driven agent is a rookie detective who follows the playbook to the letter. The RAG variants are detectives who use intuition, each with a different method for narrowing the suspect list. The stage-attributed metrics tell the chief which detective is good at finding suspects (retrieval), which one correctly eliminates red herrings (screening), and which one actually closes the case (end-to-end). The paper’s key finding: every detective can find 90% of the real suspects in the top 200 records, but *no detective can correctly eliminate the red herrings — they end up including only 52% of the true suspects after screening. The screening step is where every detective fails.

Key Concepts

• PI/ECO Criteria: A structured framework for formulating a research question in evidence-based medicine. PI/ECO stands for Population (e.g., adults with diabetes), Intervention (e.g., metformin), Comparison (e.g., placebo), Outcome (e.g., HbA1c reduction), and sometimes Study type (e.g., RCT). In the dataset, each meta-analysis question is accompanied by an explicit PI/ECO table. This is what an LLM must use to decide if a candidate study should be included. For example, if the PI/ECO says “Outcome: mortality” but a candidate study only reports quality of life, the LLM must classify it as ineligible.

• Hard Negatives: These are articles that are topically very similar to the positive studies (e.g., same disease, same intervention) but fail to meet one or more PI/ECO criteria (e.g., wrong population, wrong outcome measurement). They are the most challenging distractors for screening. In the detective analogy, these are suspects with the same modus operandi but perfect alibis. The paper shows that LLMs cannot distinguish them from positives even with the PI/ECO criteria provided.

• Stage-attributed Metrics: Instead of a single end-to-end accuracy score, this paper measures recall and precision at each stage of the pipeline: retrieval recall (how many positive studies appear in the top K retrieved documents), screening precision (what fraction of screened candidates are actually positive), and end-to-end recall (final set of included studies). This is crucial because a high retrieval recall might hide a screening failure. The key number: retrieval recall at K=200 is 90.9% (meaning almost all positive studies are retrieved), but end-to-end recall tops out at 52.7% (meaning less than half survive screening). The bottleneck is clearly screening.

Framework Shift

Before this paper, evaluation of meta-analysis automation was either end-to-end (count how many final conclusions match) or focused on retrieval only (e.g., using standard IR benchmarks). Neither approach could pinpoint the screening failure.

Before (mainstream approach):        After (this paper):
                                     +---------------------+
+---------------------+              | Stage-attributed    |
| End-to-end accuracy |              | Metrics             |
| (one number)        |              | Retrieval recall    |
+----------+----------+              | Screening precision |
           |                         | End-to-end recall   |
           v                         +----------+----------+
      "System X is Y% accurate"                |
                                      "System X has 90% retrieval       "
                                      "but only 50% screening precision"
                                      "The bottleneck is screening."

One sentence: From single-score evaluation to stage-attributed evaluation, the core shift is decomposing the pipeline to isolate the screening bottleneck.

Expert Assessment

Problem choice: Real gap. The meta-analysis community urgently needs to automate screening because it’s labor-intensive. Existing datasets only cover parts of the pipeline. This paper fills that gap cleanly.

Method maturity: Clever construction — the dataset is expert-curated from Nature journals, which gives high credibility. The baseline configurations are extensive (12 pipelines) but not exhaustive — they don’t test LLM-based screening with chain-of-thought or other reasoning techniques. The protocol-driven agent is a nice touch. The insight (screening bottleneck) is important but not surprising to anyone who has tried to build such a system; the paper confirms the difficulty with solid numbers.

Experimental integrity: Baselines are fair — they compare against standard RAG and the protocol agent. Numbers seem consistent across configurations. One red flag: the maximum end-to-end recall of 52.7% is achieved by a particular configuration, but the paper doesn’t deeply analyze *why that configuration works better. Also, the hard negatives are artificially created — they are articles that pass a topical similarity filter but fail PI/ECO. In practice, a search strategy might not retrieve them at all. The dataset’s construction might overestimate the screening challenge because the pool is already cleansed of irrelevant articles. But this is also a strength because it isolates the screening ability.

Writing quality: The paper is well-structured but dense. The related work section is thin — it could better situate MetaSyn against BioASQ, TREC Precision Medicine, etc. The abstract over-promises by saying “systematic scientific reasoning” but then only tests retrieval and screening, not statistical synthesis. The authors could have rewritten the discussion to more directly address the practical implications (e.g., how to improve screening, human-in-the-loop designs).

Verdict: weak accept — a solid benchmark that surfaces an important bottleneck, but the novelty is in the dataset rather than in the method or insights. Worth attention if you work on medical evidence synthesis or LLM evaluation.

Takeaways

• Stage-attributed evaluation is the key takeaway. If you’re building any multi-step LLM pipeline (e.g., for legal discovery, scientific literature review), measure each step separately, not just end-to-end. The screening bottleneck is not unique to meta-analysis.
• Hard negatives are a powerful evaluation tool. If you want to test whether an LLM truly understands inclusion criteria, create a pool of topically similar but ineligible documents. If the LLM can’t reject them, you know its reasoning is superficial.
• The protocol-driven agent (following exact search terms) does not outperform RAG variants in screening. This suggests that verbose search strategies are not helpful for LLM screening; a different approach (e.g., structured prompt with PI/ECO checklist) might work better. This is a concrete hypothesis to test.