Automated reproducibility assessments in the social and behavioral sciences using large language models

Paper: 2606.13670 Authors: Tobias Holtdirk, Pietro Marcolongo, Anna Steinberg Schulten, Felix Henninger, Stefan Rose, Sarah Ball, Bolei Ma, Frauke Kreuter, Markus Weinmann, Stefan Feuerriegel Categories: cs.AI

The Gap

Reproducibility in the social sciences has traditionally been evaluated by hiring independent researchers to reanalyse original data and check whether published findings hold. This approach works, but it’s painfully slow and expensive – each assessment requires weeks of human effort, domain expertise, and coordination with data curators. As a result, only a tiny fraction of published studies ever get audited, and systematic large-scale checks remain infeasible.

The paper sits at the boundary where reproducibility assessment meets automation. Prior work has attempted rule-based checks (can’t handle diverse study designs) and crowdsourced reanalysis (still human-intensive). The logical next step is to ask whether a language model that can read study descriptions, understand statistical procedures, and generate analysis code can replace the human reanalyst entirely.

[Problem: Reproducibility audits are manual, slow, and unscalable]
       |
       v
[Hypothesis: A large language model can automate the full reanalysis pipeline]
       |
       v
[Method: Feed study description (hypothesis, data, analysis plan) into LLM -> generate executable R/Python code -> compute effect size]
       |
       v
[Evidence: Test on N=76 published studies with predefined claims; compare against original results and human reanalyses]
       |
       v
[Conclusion: LLM pipeline reproduces original qualitative conclusions 96% of the time, outperforming human reanalysts (74%); effect size recovery within +/- 0.05 Cohen's d in 41% of studies]

The Increment

One sentence: Before this paper, reproducibility assessment required a human to reanalyse data – now a language model can do it automatically, opening the door to systematic auditing at scale.

Core Mechanism

The pipeline has four components. First, a study description is constructed from the original paper’s key elements: the research question, the experimental design, the variables and sample, and the exact statistical analysis reported. This description is fed into a carefully engineered prompt that instructs the LLM to act as a “reanalysis scientist”.

Second, the LLM generates analysis code (R or Python) that mirrors the original study’s procedure as closely as possible – running the same t-test, regression, or ANOVA on the same data. The prompt includes constraints like “preserve the original analytical choices” and “output the effect size (Cohen’s d) with a 95% confidence interval”.

Third, the code is executed in a sandboxed environment on the original dataset. The LLM doesn’t see the code once it’s generated; execution is done by a separate runner that captures output and errors. If the code fails, the pipeline logs a failure (7 out of 76 studies did).

Fourth, the computed effect size is compared against the original published effect size. If the absolute difference in Cohen’s d is <= 0.05, it’s a “quantitative recovery”. Separately, the pipeline reads the LLM-generated result and classifies whether it “supports the original claim” (qualitative conclusion). This dichotomous judgement is compared against the original study’s conclusion.

+----------------------------+
| Study description:         |
|  - research question       |
|  - hypothesis              |
|  - data description        |
|  - analysis plan           |
+----------------------------+
          |
          v
+----------------------------+
| LLM prompt (engineered)    |
| "Reanalyse this study."    |
| "Generate R code to       |
|  reproduce the analysis."   |
+----------------------------+
          |
          v
+----------------------------+
| Generated code (R/Python)  |
| + sandboxed execution      |
| + error handling           |
+----------------------------+
          |
          v
+----------------------------+
| Extracted effect size: d   |
| + qualitative conclusion:  |
|   does result support      |
|   the original claim?      |
+----------------------------+
          |
          v
+----------------------------+
| Compare to original:         |
|  - quantitative:              |
|    delta(d) <= 0.05?         |
|  - qualitative:               |
|    same binary verdict?       |
+----------------------------+

Structural metaphor: Think of this pipeline as a recipe replication robot in a test kitchen.

• The original study is the famous chef’s recipe card: ingredients (variables), steps (statistical tests), and final taste profile (effect size).
• The study description is a cleaned-up summary of the recipe, handed to the robot.
• The LLM is a robot chef that reads the description and writes a cooking program (analysis code).
• The sandbox execution is the robot actually cooking the dish using the exact same ingredients (data).
• The effect size is the taste score measured by a standardized scale (Cohen’s d).
• The human reanalyst is a second human chef who also tries to replicate the dish by hand, but is slower, more expensive, and sometimes misinterprets the recipe.
• The tolerance of +/-0.05 is like saying “if the taste score is within 1% of the original, we call it a perfect replication”.
• The qualitative conclusion is a simpler judgement: “does the dish still taste sweet?” rather than “exactly how many grams of sugar did the original chef use?”.

The robot chef (LLM) does surprisingly well — it gets the taste qualitatively right 96% of the time, outperforming the human chef (74%). But it only nails the exact sugar amount (effect size) in 41% of dishes (vs 34% for humans). So it’s better at the big picture than the fine detail.

Key Concepts

• Effect size (Cohen’s d): A standardized measure of the magnitude of an effect. If a treatment group has a mean 0.5 standard deviations higher than the control group, Cohen’s d = 0.5. It’s independent of sample size, so it tells you how “big” the effect is, not just “is it statistically significant”. In this paper, they use d because it’s the most common metric in social science. They set a strict tolerance of +/-0.05 — that’s a narrow window, so 41% recovery is actually non-trivial.

• Quantitative vs qualitative reproducibility: Quantitative means the exact numbers match within a tolerance. Qualitative means the *direction and significance match — e.g., both original and reanalysis say “the intervention significantly increased performance”. The authors correctly treat these as separate dimensions. A paper can be qualitatively reproducible but not quantitatively (if effect sizes differ by more than 0.05 but both say “positive effect”).

• LLM prompt engineering: The trick is not just asking the LLM to “reanalyse this study”. The authors craft a multi-part prompt that includes: (1) the exact research hypothesis, (2) the dataset structure (variables, sample size), (3) the analysis plan as described in the original paper, (4) instructions to output code that computes Cohen’s d, and (5) a constraint to “not modify the analytical procedure”. This reduces hallucination and encourages faithful reproduction.

Framework Shift

Before (human reanalysis):                 After (LLM pipeline):

[Original study]                           [Original study]
       |                                         |
       v                                         v
[Recruit domain expert]                    [Extract study description]
       |                                         |
       v                                         v
[Human reads paper,                        [LLM prompt]
 writes analysis code]                           |
       |                                         v
       v                                    [LLM generates code]
[Execute code manually,                           |
 check output]                                    v
       |                                    [Auto-execute, extract d]
       v                                           |
[Compare with original]                            v
                                     [Compare with original (auto)]
                                          |
                                          v
                                  [Scalable: 100 studies/day]

One sentence: From human-in-the-loop, slow, expensive reanalysis to LLM-in-the-loop, fast, cheap, and parallelised reproducibility assessment — the core shift is replacing expert labour with language model automation while maintaining or improving accuracy.

Expert Assessment

Problem choice: Real gap. The reproducibility crisis is well-documented, and current manual approaches are bottlenecked by human effort. An automated alternative could enable systematic audits of entire literatures, changing how we validate social science. The 76-study dataset is modest but defensible for a proof-of-concept.

Method maturity: Clever application of existing LLM capabilities, not a deep technical breakthrough. The key insight is that a well-engineered prompt can get GPT-4 or similar models to generate correct statistical code. The pipeline itself is straightforward — the novelty is in the evaluation design and the comparison against human reanalysts. Simpler approaches (e.g., rule-based extraction of reported statistics) would fail because studies vary too much. So LLMs are a reasonable choice.

Experimental integrity: Fair baselines — they compare against actual human reanalyses of these studies. The +/-0.05 tolerance is strict; one could argue it’s too narrow (many original papers might have measurement error larger than that). But they also report qualitative results, which are more important. Red flag: 7 studies were excluded because the LLM couldn’t produce a viable estimate. That’s a 9% failure rate. The authors don’t deeply analyse why these failed — likely issues with non-standard analyses, poor study descriptions, or data format problems. The human reanalysts might have also struggled with those studies, but we don’t know. Also, the qualitative agreement of 96% seems suspiciously high — could be due to ceiling effects (most social science claims are binary and the LLM might just guess “supports” most of the time). Need to see confusion matrix.

Writing quality: The abstract and method description are clear. Weakness: no discussion of computational cost (how many API calls? what model specifically? how much per study?). The code and prompts are not provided inline — relying on external repositories makes it harder to verify. The section on “failure cases” is too thin. If rewritten, I’d want a dedicated analysis of the 7 failures and a robustness test with different LLM versions.

Verdict: Weak accept — the paper validly demonstrates the concept but has enough methodological caution to avoid overselling. Has the potential to become a standard reference if followed by larger-scale studies.

Takeaways

Practitioners in any field that requires systematic reanalysis of published results (not just social science — medical trials, economics, education) can steal two things:

1. The prompt template pattern: The zero-shot prompt that says “You are a replication scientist. Given the study description below, write R code that reproduces the analysis exactly. Output the Cohen’s d and a binary conclusion.” This pattern can be adapted to any domain by swapping the statistical test and effect size metric.

2. The quantitative+qualitative evaluation framework: Don’t just check whether exact numbers match. Always report both tolerance-bound quantitative recovery and binary conclusion agreement. The latter is more robust and easier to interpret, and it’s what LLMs excel at. This dual framework can be used to benchmark any automated reanalysis system.

3. Graceful failure handling: The pipeline logs when the LLM-generated code fails to execute. That’s valuable — a real system needs to handle failure cases transparently (e.g., fallback to a human). The authors’ 9% failure rate gives a realistic baseline for system designers.

If there is nothing concrete to take? No — these are concrete, implementable patterns.