Validating LLM Simulations as Behavioral Evidence: When Can AI Replace Human Subjects?

Paper: 2602.15785 Authors: Jessica Hullman, David Broska, Huaman Sun, Aaron Shaw Categories: cs.AI

Abstract

Large language models are increasingly being used as synthetic participants in social science experiments, offering cost-effective and rapid data collection. However, the validity of using LLM simulations as evidence for human behavior remains unclear. This paper presents a framework for understanding when LLM-based simulations can support valid causal inference, distinguishing between heuristic approaches (prompt engineering, fine-tuning) suitable for exploratory research and statistical calibration methods that provide formal guarantees for confirmatory studies. The authors argue that statistical calibration, which combines auxiliary human data with statistical adjustments, can yield more precise causal effect estimates at lower cost than human-only experiments, while heuristic approaches lack the statistical rigor needed for confirmatory research.

Key Contributions

• Establishes a formal framework distinguishing exploratory versus confirmatory uses of LLM simulations in behavioral research
• Contrasts heuristic approaches (prompt engineering, model tuning) with statistical calibration methods for obtaining valid causal estimates
• Demonstrates that statistical calibration can provide formal validity guarantees and improved precision-cost tradeoffs compared to human-only studies
• Identifies critical assumptions required for each approach and clarifies when LLM simulations can substitute for human participants
• Highlights overlooked opportunities when researchers focus solely on LLM-human substitution rather than complementary uses

Two Paradigms for LLM-Based Behavioral Research

The paper identifies two fundamentally different strategies for using LLMs in behavioral studies:

Heuristic Approaches attempt to make LLM responses interchangeable with human responses through iterative refinement. Researchers employ prompt engineering, model fine-tuning, persona conditioning, and other “repair” strategies to minimize discrepancies between simulated and observed behavior. While these methods can be valuable for hypothesis generation, pilot testing, and exploratory analysis, they lack formal statistical guarantees. The validity of heuristic approaches depends on achieving sufficient similarity between LLM and human responses, but “sufficient” remains subjectively defined without a principled statistical framework.

Statistical Calibration takes a different approach by explicitly modeling the relationship between LLM simulations and human behavior. This method combines a small sample of human responses with larger-scale LLM simulations, using statistical adjustments to account for systematic discrepancies. Under explicit assumptions about the LLM-human relationship, statistical calibration preserves validity for causal inference while potentially improving precision and reducing costs compared to human-only studies. The key insight is that LLM simulations need not perfectly replicate human behavior—they must only satisfy specific statistical conditions that can be tested and validated.

Statistical Foundations and Validity Conditions

The authors formalize the conditions under which LLM simulations support valid causal inference. For a causal effect estimate $\hat{\tau}$ to be valid, the simulation process must satisfy:

$E[\hat{\tau}_{LLM}] = E[\hat{\tau}_{human}] + \epsilon$

where $\epsilon$ represents a bounded, quantifiable bias that can be estimated and corrected using auxiliary human data.

Statistical calibration achieves this through a two-stage process:

1. Collect a small human sample to estimate the LLM-human discrepancy function
2. Use this function to adjust estimates from larger LLM simulations

This approach provides several advantages:

• Formal validity: Under stated assumptions, estimates are unbiased for the target population
• Precision gains: Larger effective sample sizes from LLM simulations reduce variance
• Cost efficiency: Achieves better precision per dollar than human-only studies
• Testable assumptions: The validity conditions can be empirically evaluated

However, statistical calibration requires that the LLM-human relationship be stable and estimable from auxiliary data—assumptions that may not hold across all contexts.

Implications for Research Design

The paper challenges the prevailing focus on making LLMs perfectly mimic human participants. Instead, researchers should consider:

When to use heuristic approaches:

• Early-stage exploratory research and hypothesis generation
• Rapid prototyping of experimental designs
• Scenarios where formal statistical inference is not required
• Cases where the cost of validation exceeds the value of formal guarantees

When to use statistical calibration:

• Confirmatory research requiring valid causal inference
• Studies where precision-cost tradeoffs favor hybrid human-LLM designs
• Contexts where LLM-human relationships are stable and estimable
• Research with sufficient resources for auxiliary human data collection

Overlooked opportunities: The authors argue that myopic focus on substitution obscures other valuable uses of LLMs in behavioral research, including:

• Generating diverse experimental stimuli and scenarios
• Exploring counterfactual conditions infeasible with human participants
• Modeling heterogeneous treatment effects across subpopulations
• Augmenting rather than replacing human data collection

Takeaways

1. LLM simulations can provide valid behavioral evidence, but the appropriate method depends on research goals—heuristic approaches for exploration, statistical calibration for confirmation.

2. Statistical calibration offers a principled path to valid causal inference by combining small human samples with larger LLM simulations, potentially improving both precision and cost-efficiency.

3. Perfect LLM-human similarity is neither necessary nor sufficient for valid inference; what matters is whether specific statistical conditions hold and can be empirically validated.

4. The validity of both approaches fundamentally depends on how well LLMs approximate the target population—a question that requires empirical investigation rather than assumption.

5. Researchers should look beyond simple substitution to explore complementary uses of LLMs that augment rather than replace human participants in behavioral studies.