Hindcast: Sealing the Evidence Room to Evaluate LLM Forecasters

Paper: 2607.14051 Authors: Xiao Ye, Jacob Dineen, Evan Zhu, Shijie Lu, Kevin Song, Ben Zhou Categories: cs.CL

The Gap

Existing LLM forecasting benchmarks evaluate models through backtesting: replay resolved questions and score the probabilities a model would have assigned before the answer was known. But there are two channels through which the answer leaks into the test. First, a model with retrieval can surface news articles, forum posts, or reports written after the event — turning forecasting into a lookup task. Second, and sneakier, each new generation of models is trained on data scraped closer to the evaluation date. A question that was genuinely “in the future” for GPT-3-era models now sits inside GPT-5’s training corpus. Either way, you’re grading recall while claiming to grade foresight.

Prior work on temporal evaluation (e.g., LMSYS, future benchmarks with cutoffs) has acknowledged the training-data leak partially, but nobody has tackled both leaks simultaneously with a principled, reproducible framework. Hindcast fills this gap by designing an evaluation that stands at a chosen past date and refuses to look forward — through either channel.

Problem: LLM forecasting benchmarks leak future info
    |
    v
Two leaks: retrieval sees post-event docs,
           training data encroaches on eval questions
    |
    v
Prior work: partial fixes (cutoff dates for training,
            but no retrieval freeze)
    |
    v
Hindcast: freeze knowledge at per-question t_0,
          score vs outcome AND market price at t_0
    |
    v
Evidence: retrieval helps only when Reddit had
          pre-event discussion; hurts otherwise
    |
    v
Conclusion: temporal isolation reveals true
            forecasting ability, not memorization

The Increment

One sentence: Before this paper, evaluating an LLM forecaster was like letting a detective read today’s newspapers before solving a 1995 cold case; after Hindcast, the evidence room is sealed at a chosen date and the detective is graded against both the truth and what the original investigators concluded from the same evidence.

Core Mechanism

Hindcast’s pipeline has three stages. First, for each resolved Polymarket prediction market question, a cutoff date t_0 is chosen — a point in time before the outcome was known. Second, a frozen snapshot of public Reddit is prepared: only posts and comments timestamped before t_0 are accessible. The LLM forecaster reads this snapshot (or not, depending on the experimental condition) and produces a probability estimate for the question. Third, the forecast is scored two ways: against the actual market resolution (did the model get it right?) and against the Polymarket price at t_0 (did the model beat the crowd’s forecast made from the same information?).

The key design choices that close the leaks: the Reddit snapshot is frozen once and never updated, so no post-event information can creep in through retrieval. The cutoff is set per-market, not per-model, so a newer model trained on more data doesn’t gain an unfair advantage — it’s still being tested on questions that were genuinely unresolved at t_0. And because the snapshot never changes, the evaluation re-runs cleanly on new markets as models improve, without going stale.

The dual scoring against both the outcome and the market price at t_0 is what makes this framework genuinely useful. A model that merely matches the market price isn’t demonstrating forecasting skill — it’s demonstrating that it can read the consensus. Only models that beat the market price at t_0 are adding value beyond what humans already knew from the same information.

+-------------------+       +-------------------+
| Polymarket        |       | Reddit archive    |
| resolved question |       | (public posts)    |
+--------+----------+       +--------+----------+
         |                           |
         v                           v
+--------+---------------------------+----------+
|              Choose cutoff t_0               |
|        (before outcome was known)            |
+--------+---------------------------+----------+
         |                           |
         v                           v
+--------+----------+       +--------+----------+
| Market price      |       | Freeze snapshot   |
| at t_0            |       | to posts < t_0    |
| (human forecast)  |       +--------+----------+
+--------+----------+                |
         |                           v
         |                  +--------+----------+
         |                  | LLM reads only    |
         |                  | pre-t_0 context   |
         |                  | produces prob est |
         |                  +--------+----------+
         |                           |
         v                           v
+--------+---------------------------+----------+
|                   Scoring                     |
|  (1) model prob vs actual outcome             |
|  (2) model prob vs market price at t_0        |
+-----------------------------------------------+

Structural Metaphor

Think of Hindcast as evaluating cold-case detectives. Each case is a resolved Polymarket question. The evidence room is the Reddit archive. Normally, when we evaluate a detective on a cold case, we hand them today’s newspaper clippings along with the original case file — a newer detective who’s read more recent papers naturally “solves” more cases, but we can’t tell if they’re better at deduction or just better at reading.

Hindcast seals the evidence room on a chosen date. The detective can only examine documents filed before that date. Their solution is then scored two ways: (1) Did they identify the correct perpetrator (the actual outcome)? (2) How does their conclusion compare to the notes left by the original lead investigator, who had access to the same sealed evidence (the market price at t_0)?

This metaphor is load-bearing. The sealed evidence room = frozen Reddit snapshot (no post-event leakage). The original investigator’s notes = market price at t_0 (a calibrated human forecast from the same information). The per-case seal date = per-market cutoff (a case from 2019 is sealed at a different date than one from 2023). And the key finding maps naturally: retrieval helps the detective only when witnesses had actually discussed the case before the seal date. If the evidence room contains only speculation and rumors, retrieval hurts — it’s noise, not signal.

Key Concepts

• Information leakage in evaluation: When you test a model’s ability to predict the future, you need to make sure the model can’t see the future during the test. This sounds obvious, but it happens in two non-obvious ways. Retrieval leakage: the model’s search tool finds a Wikipedia article written after the event. Training data leakage: the model was trained on text that includes the answer. Hindcast closes both by freezing a knowledge source (Reddit) at a date before the outcome and never letting the model see anything from after that date — not through retrieval, and not through training (by choosing t_0 dates that are genuinely in the model’s past).

• Market price as baseline: Instead of just checking “did the model get the right answer?”, Hindcast also checks “did the model beat the crowd?” The Polymarket price at t_0 is itself a forecast — a probability estimate made by humans who had access to the same pre-t_0 information. If a model merely matches this price, it’s doing no better than a reader of the same Reddit threads. Only beating the market price demonstrates genuine forecasting edge. This is like grading a student not just on correctness, but on whether they outperformed the class average on the same exam.

• Temporal isolation: The core discipline that makes Hindcast work. Every component of the evaluation — the knowledge source, the scoring baseline, the question selection — is anchored to a single point in time t_0. Nothing from after t_0 is allowed in. This makes the evaluation reproducible and fair across model generations, because the question “could this model have known?” has a crisp, verifiable answer.

Framework Shift

Before (mainstream approach):              After (Hindcast):

Q + "all available knowledge"             Q + frozen snapshot at t_0
         |                                          |
         v                                          v
     LLM forecast                             LLM forecast
         |                                          |
         v                                          v
  Score vs outcome only                    Score vs outcome
                                           AND vs market@t_0

  Leaks: retrieval + training              No leaks: both sealed
  Baseline: none / accuracy                Baseline: crowd wisdom
  Stale: yes (new models break it)         Stable: re-runs cleanly

From “grade against the answer” to “grade against the answer and the crowd’s forecast from the same frozen moment,” the core shift is treating evaluation as a time-travel problem where every information channel must be sealed at the same date.

Expert Assessment

Problem choice: This is a real gap, not manufactured. The dual-leakage problem — retrieval and training data — is underappreciated in the forecasting evaluation literature. Most temporal benchmarks focus on training data cutoffs and ignore retrieval entirely. Hindcast addresses both simultaneously, and the prediction-market framing gives it a natural, economically grounded baseline. This sits squarely in the trajectory of “making LLM evaluation trustworthy,” which is one of the field’s most important open problems.

Method maturity: Mostly clever insight rather than brute force. Using prediction markets as both question source and baseline is elegant — markets are liquid, resolved, and come with calibrated probability prices. The frozen Reddit snapshot is a pragmatic choice (public, timestamped, large), though it introduces a domain bias: Reddit skews toward certain topics and demographics. The per-market t_0 cutoff is the key engineering contribution that prevents the evaluation from going stale. A simpler approach might use a single global cutoff, but the per-market design is strictly better and worth the complexity.

Experimental integrity: The dual-scoring against market prices is a genuine contribution to baseline fairness — most forecasting benchmarks have no baseline at all, just raw accuracy. The finding that retrieval hurts when Reddit only had speculation is both plausible and important. One concern: Reddit is not the only public knowledge source, and results may not transfer to models using different retrieval corpora. The authors should be explicit about this limitation. Also, Polymarket questions skew toward politics, crypto, and pop culture — generalizability to other forecasting domains (scientific, geopolitical) is untested.

Writing quality: The abstract is exceptionally clear — it identifies the problem, the mechanism, and the key finding in three tight sentences. From what’s visible, the writing is disciplined. The weakest spot (likely) is the related work section: the paper would benefit from a more systematic comparison to temporal evaluation methods in NLP beyond forecasting (e.g., temporal fact verification, time-aware QA). A table mapping prior work’s leakage coverage would elevate the framing.

Verdict: weak accept — The problem is real, the solution is principled, and the key finding (retrieval helps only with pre-event discussion) is genuinely useful for practitioners designing forecasting systems. The scope is narrow (Reddit + Polymarket only), but that’s appropriate for a first cut at this problem.

Takeaways

1. When building any “predict the future” benchmark, audit for both retrieval and training-data leakage. Most teams only think about one. Hindcast gives you a concrete checklist: freeze your knowledge source at a per-question cutoff, and score against a crowd baseline from the same frozen moment.

2. Retrieval is not universally helpful for forecasting — it depends on whether the corpus actually discussed the event beforehand. If you’re building a retrieval-augmented forecaster, measure retrieval quality separately: how much of the relevant information existed in your corpus before the cutoff? This finding transfers directly to any RAG system evaluated on temporal tasks.

3. Prediction markets are an underused source of calibrated human baselines. If you’re evaluating any system that produces probabilities, check whether a prediction market existed for the same question. The market price at t_0 is a free, high-quality baseline that tells you whether your model is adding value beyond consensus.