Autonomous AI Systems Generate Expert-Level Disease Forecasts

Paper: 2605.16238 Authors: Sarah Martinson, Michael P. Brenner, Martyna Plomecka, Brian P. Williams, Nicholas G. Reich, Zahra Shamsi Categories: cs.AI

The Gap

Disease forecasting teams manually craft models for each pathogen and location. This expert-driven process works but doesn’t scale: when RSV surges in a new county or a novel pathogen emerges, you need epidemiologists to design models from scratch. The CDC hub aggregates human-curated models, but the curation bottleneck limits geographic granularity and response speed. Prior automated forecasting (statistical baselines, neural nets) either lacks interpretability or fails to incorporate domain theory, making them unsuitable for public health deployment where transparency matters.

Problem: Manual model curation bottleneck
   |
   v
Assumption: Epidemiological theory can be
            translated to code algorithmically
   |
   v
Method: LLM-guided tree search generates
        + evaluates + optimizes models
   |
   v
Evidence: Prospective 2025-26 evaluation
          matches CDC expert ensembles
   |
   v
Conclusion: Autonomous forecasting at scale
            without sacrificing accuracy

The Increment

One sentence: Before this paper, disease forecasting required expert teams to manually code models for each pathogen-location pair; after, an autonomous system generates comparable forecasts by searching the space of executable epidemiological models.

Core Mechanism

The system operates as a code-generating search process. An LLM proposes candidate forecasting models as executable Python code, guided by epidemiological theory from its training. Each candidate is evaluated on historical data using a log-scale distance metric. The search tree expands by mutating high-performing branches—refining data preprocessing, adjusting model structure, or trying different statistical approaches. An automated judge verifies that generated code actually implements the intended epidemiological mechanism (e.g., checking that an SEIR model truly has susceptible-exposed-infected-recovered compartments). The search runs until performance plateaus, then the top-k diverse models form an ensemble.

   [Epidemiological Theory]
            |
            v
    +---------------+
    |  LLM Proposer |  --> Generates Python code
    +---------------+      (ARIMA, SEIR, ensemble, etc.)
            |
            v
    +---------------+
    | Eval on Data  |  --> Log-scale distance metric
    +---------------+
            |
            v
    +---------------+
    | Judge Check   |  --> Structural fidelity test
    +---------------+
            |
            v
    Tree expansion: mutate high-performers
            |
            v
    Final ensemble of top-k models

Think of this as evolutionary recipe development in a test kitchen. The LLM is a chef who knows culinary theory (epidemiology) and proposes recipes (models). Each recipe gets cooked (executed on data) and tasted (evaluated). A food critic (judge) checks whether the dish actually follows the intended technique—if you claimed to make a soufflé, did it rise? The kitchen keeps the best recipes, tweaks them (tree search mutations), and tries variations. Eventually, you serve a tasting menu (ensemble) of the top dishes. The key insight: you don’t need the head chef to manually write every recipe if you have a system that can generate, test, and refine them automatically while respecting culinary principles.

Key Concepts

• Log-scale distance metric: Standard forecasting metrics (mean absolute error, squared error) measure distance in absolute case counts. If a model predicts 1000 cases when truth is 1100, that’s an error of 100. But if it predicts 10 when truth is 110, that’s also 100—yet the second is a 10x mistake while the first is 10%. Log-scale metrics measure *relative error by comparing log(prediction) to log(truth). This prevents “reward hacking” where models game the metric by always predicting high numbers (large outbreaks dominate absolute error, so predicting big is safe). In epidemiology, getting the order of magnitude right matters more than exact counts, especially during exponential growth phases.

• Judge-in-the-loop: The LLM can generate syntactically correct code that doesn’t actually implement the claimed epidemiological model. For example, it might output an SEIR model that skips the “exposed” compartment or uses the wrong differential equations. The judge is an automated verifier (itself an LLM with a specialized prompt) that reads the generated code and checks structural properties: Does this SEIR model have four compartments? Does transmission depend on susceptible-infected contact? Does it integrate the ODEs correctly? This catches “specification drift” where the search optimizes for metric performance by accidentally abandoning the theoretical framework.

Framework Shift

Before (mainstream approach):        After (this paper):

Expert Team                          LLM + Tree Search
    |                                     |
    v                                     v
Manual model design                  Auto-generate candidates
    |                                     |
    v                                     v
Code implementation                  Auto-evaluate on data
    |                                     |
    v                                     v
Validation + tuning                  Judge structural fidelity
    |                                     |
    v                                     v
Deploy for one pathogen              Ensemble top performers
                                          |
[Repeat for each new                      v
 pathogen/location]                  Deploy for all pathogens

Bottleneck: Human labor              Bottleneck: Compute budget

From artisanal model crafting to automated model discovery, the core shift is replacing human iteration with algorithmic search over the space of executable theories.

Expert Assessment

Problem choice: Real and well-motivated. The CDC hub exists precisely because manual forecasting doesn’t scale, and the 2025-26 prospective evaluation proves the problem isn’t hypothetical. The timing is good—LLMs are now capable enough to generate non-trivial scientific code, and disease forecasting has standardized evaluation infrastructure (FluSight, COVID-19 Forecast Hub).

Method maturity: The tree search + judge architecture is sound, but the heavy lifting happens inside the LLM’s ability to propose sensible models. The paper doesn’t deeply ablate what makes a good proposal distribution—how much comes from epidemiological pretraining vs. in-context examples vs. search dynamics? The log-scale metric fix is clever and well-justified. The judge-in-the-loop feels necessary but underexplored: how often does it catch real violations vs. false positives? What happens when the judge and the proposer are the same model family?

Experimental integrity: The prospective evaluation is the paper’s strongest asset—no hindsight bias, real-time comparison to CDC ensembles. The RSV “cold start” scenario is a good stress test. However, the baselines are somewhat limited: they compare to CDC hub ensembles (which aggregate human models) but not to other automated approaches like pure neural forecasters or simpler LLM-based methods without tree search. The ablations (log-scale metric, judge) are informative but narrow. I’d want to see: what if you just prompted GPT-4 to write one good model without search? How much does ensemble diversity matter?

Writing quality: The abstract and introduction are crisp. The methods section buries important details (judge prompt design, tree search hyperparameters) in ways that hurt reproducibility. The results focus heavily on aggregate performance but don’t show failure modes—when does the system generate nonsense? The discussion oversells “unprecedented scales” when the evaluation covers three pathogens in one country. A rewrite of the methods section with clearer algorithmic pseudocode and judge examples would elevate the paper significantly.

Verdict: weak accept — Solid prospective results and a real problem, but the method feels like “LLM + standard search + domain-specific fixes” rather than a fundamental innovation. The contribution is more engineering integration than algorithmic novelty, which is valuable but not groundbreaking.

Takeaways

Steal the log-scale evaluation trick: If your domain has exponential dynamics (finance, network growth, viral spread), standard error metrics let models game the system by predicting conservatively high. Switching to log-space forces models to get the growth rate right, not just the peak magnitude.

Judge-in-the-loop for scientific code generation: When generating code that must satisfy domain constraints (physical laws, statistical properties, algorithmic correctness), add a verification step that checks structural properties beyond “does it run?” This prevents the generator from finding shortcuts that optimize metrics while violating theory.

Ensemble diversity from search trees: Instead of training multiple models with different random seeds, use tree search to explore methodologically different approaches (ARIMA vs. mechanistic vs. ensemble) and aggregate them. The search naturally finds diverse high-performers if you penalize redundancy.

Cold start strategy: The RSV scenario shows that even with sparse historical data, the system can bootstrap by generating simple models (moving averages, naive baselines) and iteratively complexifying. This “start simple, search for complexity” approach transfers to any domain where you need to forecast a new phenomenon with limited data.