Fisher-R1: Teaching Agents That a Correct p-Value Can Still Be Wrong

Paper: 2608.07437 Authors: Jiacheng Miao, Jin Mu, Guanhua Chen, James Zou Categories: cs.AI

The Gap

Data-analysis agents have gotten good at the visible part of the job. Give one a CSV and a question, and DS-1000, DABench, DiscoveryBench, BLADE and friends will tell you whether it loaded the file, wrote runnable pandas, called the right library function, and produced a number that matches a reference answer. Execution accuracy is largely solved for tabular work.

What none of those benchmarks really score is whether the number means what the agent says it means. A ttest_ind call can execute perfectly on data with repeated measures per subject, clustered sampling, wild heteroskedasticity, or twenty hypotheses tested in sequence — and return a p-value that is simply not a valid p-value for the claim being made. The code is green. The inference is broken. Because reference-answer benchmarks reward matching a number, an agent that reaches 0.03 by the wrong route scores identically to one that reaches 0.03 legitimately, and an agent that correctly reports 0.21 after clustering the standard errors can be marked wrong.

This paper’s claim is that this is the dominant remaining failure mode for scientific-analysis agents, that it is invisible to current evals, and that it is trainable. Hence two artifacts: P-Bench (425 open-ended hypothesis-testing tasks in economics, biology, medicine, where the agent gets only a hypothesis and a dataset and must choose a method, compute a p-value, and state a conclusion) and Fisher-R1 (an open-weight agent RL-trained on synthetic tasks whose statistical ground truth is known by construction).

   [ PROBLEM ]  agents produce correctly executed but invalid tests
        |
        v
   [ ASSUMPTION ]  validity is checkable when you know the
                   data generating process
        |
        +----------------------------+
        |                            |
        v                            v
   [ METHOD A ]                 [ METHOD B ]
   P-Bench: 425 open-ended      synthetic DGPs with known
   tasks scored on              correct test  ==>  verified
   inferential validity         statistical reward  ==>  RL
        |                            |
        +-------------+--------------+
                      v
   [ EVIDENCE ]  frontier agents stumble on P-Bench.
                 Fisher-R1-14B  >  backbone
                 Fisher-R1-14B  >  GPT-5.4 / DeepSeek-V4-Pro
                 +21% relative single-trial success
                 +26% on hardest split
                      |
                      v
   [ CONCLUSION ]  statistical rigor is a learnable skill
                   not an emergent property of scale

The Increment

One sentence: Before, we measured whether an analysis agent’s code ran and its number matched; after, we measure whether its p-value is *licensed by the data — and we have a 14B open model that does this better than frontier proprietary agents.

Core Mechanism

Two halves, and the interesting one is the training side. P-Bench is the diagnostic: open-ended tasks with no reference number handed to the agent, graded on the joint object (method choice, p-value, conclusion). “Open-ended” is doing real work here — the moment you supply a reference p-value, you’ve smuggled in the method, and the failure mode you wanted to measure disappears from the test.

Fisher-R1 attacks the training-signal problem. You cannot get a trustworthy reward from real datasets, because on real data nobody knows the truth — that is the entire reason hypothesis testing exists. So the authors invert it: generate the task from a known data generating process. You decide there are within-subject repeated measures. You decide the variance scales with the covariate. You decide the true effect is zero. Now the set of valid analyses is determined by construction, and so is the correct conclusion. A reward function can check the agent’s chosen method against the assumptions you injected, check the p-value it computed, and check whether the verbal conclusion follows. That reward is verifiable in the RLVR sense — no LLM judge guessing at rigor.

Then it’s a rollout loop: sample a synthetic task, let the agent inspect the data and write and run code in a sandbox, extract the method-p-value-conclusion triple, score it against the generator’s ground truth, and do policy optimization on the reward. Repeat across a spread of assumption violations so the model stops reaching for ttest_ind as a default reflex and starts checking the shape of the data first. Evaluation then happens on P-Bench, which is a different distribution of tasks.

  TRAINING SIDE                          EVAL SIDE
  =============                          =========

  [ DGP sampler ]                        [ P-Bench ]
  pick: clustering                       425 tasks
        heteroskedasticity               econ / bio / med
        repeated measures                only: hypothesis
        multiplicity                           + dataset
        true effect = 0 or d
        |
        | emits both
        +-------------------+
        |                   |
        v                   v
  ( dataset )        ( ground truth:
  ( + prompt )         valid methods
        |              + true answer )
        v                   |
  [ AGENT rollout ]         |
   inspect data             |
   write code               |
   run in sandbox           |
   report triple:           |
    method / p / claim      |
        |                   |
        v                   v
     +---------------------------+
     |  VERIFIED STAT REWARD     |
     |  method fits assumptions  |
     |  p-value correct          |
     |  conclusion follows p     |
     +---------------------------+
                |
                v
        [ policy update ]  --loop-->  Fisher-R1-14B  ==>  scored on P-Bench

The metaphor: this is a flight simulator for statisticians.

You cannot grade a pilot by watching them fly real weather, because you don’t know what the wind was actually doing. So you build a simulator where you set the crosswind, the icing, the failed hydraulic line. The DGP sampler is the scenario generator: clustering is the crosswind, heteroskedasticity is the icing, multiple comparisons is a slow fuel leak that only kills you if you ignore it. Because you authored the weather, you know exactly what the correct control input was — that’s the verified reward, an automatic instructor rather than a human squinting at the tape. The agent’s sandboxed code execution is the pilot’s hands on the yoke. Thousands of simulator hours is the RL loop, which is how a reflex (“this looks paired, don’t use an independent-samples test”) gets built rather than recited. P-Bench is the check ride: real-feeling routes the trainee never practiced, flown for a grade.

And the old benchmarks? They graded whether the plane landed at the right airport. Which it can, after you flew straight through a thunderstorm you should have gone around.

Key Concepts

• Validity vs. correctness of a p-value: A p-value is a promise, not a number. The promise is: *if there were truly no effect, I would see a value this small at most 5% of the time. That promise only holds if the recipe’s assumptions match the data. Concretely: measure blood pressure twice on each of 50 patients, then run an independent two-sample t-test on the 100 rows as if they were 100 people. Your code is correct — the function computed exactly what it advertises. But you told it you had 98 degrees of freedom when you had roughly 49, so the promise is broken: under the null you’ll get p < 0.05 far more than 5% of the time. The number is correct; the inference is invalid. Every existing eval sees a valid-looking float and moves on.
• Verified reward from a generative process: The usual bind in RL for reasoning is that you need a grader. For math you have the answer key; for open-ended analysis you don’t. The escape hatch: don’t find tasks, *manufacture them, so the answer key is a byproduct of construction. If you wrote the simulation that put clustered noise in the data, you know clustered standard errors are required and you know the true effect is zero. The reward is a program, not a judgment call. The cost is distribution shift — your simulator is not the world — which is exactly why P-Bench has to exist separately.
• Open-ended task design as a measurement instrument: Whether a benchmark can detect a failure mode is a property of its *format, not its difficulty. Supply a reference p-value and the agent can back out the intended method; grade only the final number and you cannot distinguish luck from rigor. P-Bench’s design decision — hypothesis plus raw data, nothing else, graded on the whole triple — is what makes the failure mode observable at all. That is the transferable move.

Framework Shift

Before (mainstream approach):          After (this paper):

  task + reference answer                task = hypothesis + raw data
        |                                      |
        v                                      v
  [ agent writes code ]                  [ agent inspects data ]
        |                                      |
        v                                      v
  [ does it run    ]                     [ what are the
  [ does the number]                     [ assumptions here ]
  [ match          ]                           |
        |                                      v
        v                                [ pick method that
     PASS / FAIL                         [ survives them
     ~~~~~~~~~~~                               |
     execution                                 v
     accuracy                             [ p-value + claim ]
                                                |
                                                v
                                        graded on VALIDITY
                                        ~~~~~~~~~~~~~~~~~~
                                        was the number licensed

  reward source: answer key              reward source: known DGP
  trained by: SFT on traces              trained by: RL on verified
                                                     stat reward
  failure mode measured:                 failure mode measured:
    wrong code                             right code / wrong inference

From grading the artifact to grading the argument, the core shift is that correctness of computation and validity of inference are different variables, and only the second one is science.

Expert Assessment

Caveat up front: I’m working from the abstract and the framing, not a full read of the appendix, so treat the experimental critiques as things to check rather than things I’ve confirmed.

Problem choice: Real gap, and well-placed. The AI-for-science agenda has moved from “can the agent write code” to “can I put this in a paper,” and the bottleneck is exactly the class of error that leaves no trace in the logs. It’s also the *right narrow slice — hypothesis testing has crisp, checkable validity conditions, unlike, say, “was this causal claim reasonable.” Slightly manufactured in one respect: the claim that existing benchmarks “fail to capture” this is a bit strong, since BLADE and some DiscoveryBench splits do probe analysis-decision quality. The honest version is “existing benchmarks don’t isolate or systematically stress it.”

Method maturity: The recipe (synthetic tasks + verifiable reward + RL) is 2025-standard; there is nothing new in the optimization. The insight worth the paper is recognizing that statistics is an unusually good fit for RLVR, because the ground truth is not scraped or annotated but *authored — you get an infinite, difficulty-controllable task supply for free. That’s genuinely clever framing rather than brute force. What I’d want ruled out: the obvious cheaper baseline is scaffolding, not training. Give GPT-5.4 an assumption-diagnostics toolchain plus a mandatory pre-registration step (“state your assumptions and how you’ll check them before writing the test”) and a self-critique pass. If a 14B RL model beats a naked frontier model but loses to a well-scaffolded one, the contribution is much smaller. I’d bet the paper compares against agents with only light scaffolding.

Experimental integrity: 425 tasks across three fields is thin once you split by difficulty — the “+26% on the hardest tasks” number is probably resting on a couple hundred items at most, and I’d want confidence intervals and multi-seed variance before believing the ordering against frontier models. Reporting *relative single-trial improvement (21%) without foregrounding absolute rates is a mild rhetorical choice; 21% relative on a 40% base is a very different story from 21% on an 80% base. The biggest risk is a shared-ancestry leak: if P-Bench tasks and the RL training tasks come from overlapping generator logic or the same assumption taxonomy, Fisher-R1 is being tested on its training distribution while baselines are not. The paper needs an explicit generator-disjointness argument and, ideally, a held-out set of tasks from real published analyses. Minor smell: the abstract writes both “DeepSeekV4-Pro” and “DeepSeek-V4-Pro,” which suggests the numbers section hasn’t been proofread as carefully as one would like.

Writing quality: The abstract is efficient and the framing is sharp. The corner-cutting is almost certainly in how validity is *scored — “correct method given the assumptions” is not a single well-defined target, because several methods are often defensible (cluster-robust SEs vs. mixed model vs. aggregating to subject means). If the grader accepts one canonical answer, the benchmark penalizes competent analysts, and that would be the section I’d rewrite: a precise, worked-out account of the equivalence classes of acceptable analyses and the inter-rater agreement on them. Done well, that turns a nice paper into the reference artifact everyone cites.

Verdict: weak accept — the failure mode is real and the benchmark is the durable contribution, but the training recipe is off-the-shelf and the headline “beats GPT-5.4” claim rests on a small benchmark whose independence from the training generator is the whole ballgame.

Takeaways

Concrete things to steal:

• Split your eval axes: execution correctness vs. output validity. This generalizes far past statistics. A SQL agent can produce a running query that answers a subtly different question; a finance agent can compute a correct IRR on the wrong cash-flow convention. If your eval only checks “did it run and match,” you are blind to your production failure mode. Add a validity axis with its own score.
• When you lack an answer key, author the world instead of labeling it. Any domain with a simulatable generative process — A/B testing, causal inference, survey weighting, queueing, backtesting, forecasting — can produce unlimited RL tasks with free, program-checkable ground truth and a difficulty dial. The pattern is: sample the latent structure first, emit the observable data second, keep the structure as the reward key.
• Strip the reference answer to expose reasoning. Handing the model a target number leaks the intended method. If you want to measure judgment rather than recall, give the minimum viable input and grade the whole decision chain, not the terminal value.
• The assumption-violation taxonomy is a free lint checklist. Even without touching the model, you can hard-code a pre-analysis gate in your own agent: check for repeated measures / nesting, unequal variances, non-normality with small n, and how many tests have already been run this session. Most real-world agent inference errors are in that short list.
• Small-model-plus-narrow-RL beats frontier-general is now a repeatable play. If you have a domain where reward is programmatically verifiable, a 14B trained model is a plausible cheaper substitute for a frontier API call. Worth pricing out for any high-volume, narrow analysis pipeline.