AutoSynthesis: Can AI Agents Replace the Entire Meta-Analysis Pipeline?

Paper: 2607.15247 Authors: Moein Taherinezhad, Sebastian Maier, Gerardo Vitagliano, Francesco Pierri, Stefan Feuerriegel Categories: cs.AI

The Gap

Meta-analysis is the gold standard for synthesizing quantitative evidence across studies—it’s how medicine, education, and policy turn scattered findings into actionable knowledge. But the pipeline is painfully manual: searching databases, screening hundreds of abstracts, extracting statistics from PDFs, computing standardized effect sizes, fitting random-effects models, assessing bias. Each step demands domain expertise and weeks of human labor. Prior work has automated individual pieces—tools like ASReview or Rayyan assist with screening, and various extraction aids exist—but nobody has stitched the entire workflow into a single, end-to-end system that takes a natural-language research question and outputs a PRISMA-aligned meta-analytic report. AutoSynthesis fills that gap: it’s the first multi-agent system that closes the loop from question to pooled effect estimate without human handoffs.

Manual meta-analysis bottleneck
  |
  v
Existing tools automate isolated steps (screening, extraction)
  |
  v
Gap: no end-to-end pipeline from question to report
  |
  v
AutoSynthesis: multi-agent system covers all stages
  |
  v
Evidence: 28 studies, 20+ claims, Hedges' g ~ expert results
  |
  v
Conclusion: automated meta-analysis is feasible and scalable

The Increment

One sentence: Before this paper, meta-analysis was a human marathon; after it, a single prompt can produce a transparent, statistically sound evidence synthesis from scratch.

Core Mechanism

AutoSynthesis is built as a pipeline of specialized LLM agents, each owning one stage of the meta-analysis workflow. The orchestrator receives a research question in plain English, then delegates sequentially. First, a search strategy agent translates the question into Boolean queries for academic databases (e.g., PubMed, Semantic Scholar). A retrieval agent executes those queries and collects candidate papers. Two independent screening agents evaluate titles and abstracts for inclusion, mimicking the dual-reviewer standard in systematic reviews. A full-text eligibility agent then reads the surviving papers and makes the final include/exclude decision. From the included set, a data extraction agent parses full texts to pull out quantitative statistics—sample sizes, means, standard deviations, test statistics. An effect-size computation agent converts these raw numbers into standardized effect sizes (e.g., Hedges’ g). A meta-analysis agent fits a random-effects model to pool estimates across studies. Optionally, a heterogeneity agent examines moderators (study design, population, etc.) and a risk-of-bias agent evaluates methodological quality. Finally, a reporting agent assembles everything into a structured report following PRISMA guidelines, complete with forest plots and summary tables.

[Research Question (NL)]
        |
        v
[Search Strategy Agent] --> Boolean queries
        |
        v
[Retrieval Agent] --> candidate papers
        |
        v
[Screening Agents x2] --> include/exclude (abstract)
        |
        v
[Eligibility Agent] --> include/exclude (full-text)
        |
        v
[Data Extraction Agent] --> raw statistics
        |
        v
[Effect Size Agent] --> Hedges' g, etc.
        |
        v
[Meta-Analysis Agent] --> pooled estimate (RE model)
        |
        v
[Heterogeneity + RoB Agents] --> moderator analysis
        |
        v
[Reporting Agent] --> PRISMA-aligned report

Think of AutoSynthesis as a virtual research lab assembled on demand. The PI (orchestrator) gets a grant proposal—your research question. She hires a librarian (search agent) who knows exactly which databases to query and how. Two independent research assistants (screening agents) sit side by side, each reviewing abstracts and flagging disagreements for the senior RA (eligibility agent) to resolve by reading full texts. A data-entry team (extraction agent) transcribes every relevant number from the papers. The lab’s statistician (meta-analysis agent) takes those numbers, standardizes them, and fits a random-effects model. A quality auditor (risk-of-bias agent) checks whether the studies are methodologically sound. And finally, the report writer (reporting agent) drafts the paper in PRISMA format—figures, tables, the whole thing. The key insight: just as a lab needs each specialist to produce a credible meta-analysis, AutoSynthesis needs each agent to produce a credible automated one. Skip the librarian and you get a biased search. Skip the statistician and you get raw numbers without synthesis. The metaphor is load-bearing because it reveals why the pipeline must be multi-agent: no single LLM call can wear all these hats simultaneously with the required rigor.

Key Concepts

• Meta-analysis: Imagine you’ve run 30 separate experiments testing whether a new teaching method improves test scores. Each study reports a slightly different effect size. Meta-analysis is the statistical technique that combines all 30 into one weighted average, where larger, more precise studies get more influence. The “random-effects” part means we assume each study’s true effect may vary slightly—some populations respond more than others—so we estimate both the average effect *and the spread. Without meta-analysis, you’re stuck arguing about which single study to trust.

• Multi-agent system: Instead of asking one AI model to do everything at once (which degrades quality on complex tasks), you split the job into specialized roles. Each agent has its own prompt, its own context window, its own evaluation criteria. The orchestrator passes outputs from one agent as inputs to the next—like an assembly line where each station adds value. The advantage over a monolithic prompt: each agent can be optimized, debugged, and validated independently.

• PRISMA guidelines: Think of PRISMA as the formatting requirements for a legal brief, but for evidence synthesis. It mandates specific sections (search strategy, inclusion criteria, risk-of-bias assessment, forest plots) so that any reader can reproduce the analysis. AutoSynthesis’s commitment to PRISMA output is what makes it more than a toy—it produces something a journal reviewer could actually evaluate.

Framework Shift

Before (mainstream approach):          After (this paper):

  Human PI                              NL question
    |                                      |
    v                                      v
  Manual search                        Search Agent
    |                                      |
    v                                      v
  Manual screening                     Screening Agents x2
    |                                      |
    v                                      v
  Manual extraction                    Extraction Agent
    |                                      |
    v                                      v
  R/Python by hand                     Meta-Analysis Agent
    |                                      |
    v                                      v
  Word doc (maybe PRISMA)              PRISMA Report
    |                                      |
  Weeks to months                      Hours to minutes

From a human-intensive, weeks-long research project to a single-prompt automated pipeline, the core shift is replacing the research team with coordinated LLM agents at every stage of the evidence synthesis workflow.

Expert Assessment

Problem choice: This is a real gap. Meta-analysis is genuinely bottlenecked by human labor—Cochrane reviews routinely take 1-2 years. The prior art automates pieces (screening tools, extraction helpers) but nobody has closed the loop end-to-end. The timing is right: LLMs are now good enough at reading scientific text to make this plausible. This sits squarely at the frontier of AI-for-science.

Method maturity: It’s clever orchestration, not a fundamental algorithmic breakthrough. The architecture is essentially a chain of prompt-engineered LLM calls with domain-specific instructions at each stage. There’s nothing wrong with that—sometimes the engineering *is the contribution—but researchers expecting a novel statistical method or training procedure will be disappointed. Simpler approaches (a single well-prompted GPT-4 call with retrieval-augmented generation) might get 80% of the way there for many use cases, though with less auditability.

Experimental integrity: This is where I’d push back hardest. Twenty-eight studies and 20+ extracted claims is a small demonstration, not a rigorous evaluation. The claim that pooled Hedges’ g values are “similar” to expert analyses is promising but vague—what’s the mean absolute deviation? How do confidence intervals compare? Did they test on questions where the true answer is well-established *and on contested or ambiguous topics? A single-case meta-analysis (one research question) would be more convincing if deep; a breadth play with 28 studies feels preliminary. I’d also want to see failure mode analysis: where does the system hallucinate statistics, misclassify studies, or produce misleading forest plots?

Writing quality: The abstract is clean and well-structured, following a standard problem-method-results arc. The PRISMA alignment claim is smart—it immediately signals rigor. However, the abstract reads more like a system demo than a research contribution; the “what’s new in AI methodology” story is undersold. Section 4 (if it exists) comparing AutoSynthesis outputs to ground truth across multiple meta-analyses would make or break the paper.

Verdict: weak accept — a practically useful engineering contribution that demonstrates feasibility, but with evaluation too thin to be definitive and limited novelty in AI methodology itself.

Takeaways

1. The multi-agent decomposition pattern transfers directly. If you’re building any complex analytical pipeline (systematic review, patent landscape analysis, competitive intelligence), the architecture here—specialized agents with explicit handoffs—is a reusable template. The key design choice is that each agent gets a narrow, well-defined task with clear input/output contracts.

2. PRISMA as an auditability scaffold. For any AI system that produces evidence-based outputs, adopting a human-designed reporting standard (PRISMA, CONSORT, etc.) is a cheap way to make the system’s reasoning transparent and reviewable. This framing—“AI output should meet existing quality standards”—is more practical than inventing new evaluation frameworks.

3. Dual-screening with LLMs is a clever trick. Running two independent screening agents and flagging disagreements (mimicking dual human review) is a simple but effective way to catch errors without ground truth. This “LLM ensemble for quality control” pattern is worth stealing for any classification or judgment task in a pipeline.