Phantom Gains: Your Self-Improvement Numbers Are Probably Batching Artifacts

Paper: 2608.20290 Authors: Cheng Xu, Nan Yan, Liming Chen, M-Tahar Kechadi Categories: cs.AI, cs.CL

The Gap

The self-improvement literature has quietly changed what it measures. Instead of reporting mean accuracy, papers increasingly report which individual problems a model gained and which it lost. That switch is well motivated: a model that trades one set of failures for another while the average stays flat has genuinely changed, and only transition-level accounting can see it.

But a transition is the difference between two noisy estimates. Differencing amplifies measurement noise, so transition-level claims are structurally more fragile than mean-accuracy claims — and the field moved to this metric without moving to the corresponding discipline of measuring a null. There is no established practice of asking: what would this gain statistic report on a model that never trained at all?

The authors audit that question directly and find the answer is alarming. Seven distinct measurement failures are identified, and the damning property is that each one inverts a reported finding when its control is absent. Several of the seven are standard practice, not sloppiness.

   [METRIC SHIFT IN THE FIELD]
   mean accuracy  ->  which problems were gained / lost
        |                        |
        |                        +-- differences two noisy estimates
        |                        +-- noise is amplified, not cancelled
        |                        +-- needs a measured null to interpret
        v
   [GAP] The field adopted the difference
         without adopting the null
                |
                v
   Audit: three LoRA self-training rounds vs a frozen
          control pushed through the identical pipeline
                |
                v
   SEVEN measurement failures, each inverting a finding
   when the control is dropped

The Increment

One sentence: Before this paper, transition-level self-improvement claims were made with no null model to compare against; after it, the authors show seven standard practices that manufacture improvements out of pure noise — and give one exact test that correctly reports nothing on an untrained model.

Core Mechanism

The experimental setup is deliberately boring, and that is the point. Three rounds of rank-32 LoRA self-training on Qwen3-8B, alongside a frozen control pushed through the identical pipeline. The control is the whole method: it is a model that never trains, so any statistic that reports a change for it is reporting an artifact.

The failures the audit turns up are concrete. A ledger built on a single greedy decode manufactures capability changes on the untrained model, largely an artifact of inference batching — the harness reorders and groups prompts, and that alone moves per-problem outcomes around. The expansion statistic, widely used to separate genuine acquisition of new capability from mere sharpening of existing capability, assigns that same untrained model a rate of 0.280. The natural repair — put a threshold on the statistic — does not survive replication: estimated across the frozen comparisons the design already contains, its null stays non-zero. A threshold calibrated on a null that is itself non-zero is not a fix.

The replacement is a per-problem exact test against a pooled baseline under false-discovery-rate control. Its three properties are the point: it detects nothing on any held-out replicate that should show nothing; it is unchanged under the choice of multiple-testing rule, error rate and pool size; and it is stable enough to be trusted. Applied to a ladder of arms matched in stream, volume and evaluation, the audit then produces a real finding: external distillation improves problems the base model rarely reaches, while three forms of self-training do not. Even that is then stress-tested — a regression rejects the asymmetry as a by-product of distillation’s larger overall gain, with p under 10^-8.

The closing caveats are the honest part. On the far smaller set of problems the base model never reaches at all, the evidence is inconclusive. And self-training corrupts problems solved at baseline at rates well above the measured floor, which is a harm the mean-accuracy framing would have hidden.

   THE AUDIT LADDER
   [Qwen3-8B base]
        |
        +--> FROZEN CONTROL (never trains, identical pipeline)
        |         |
        |         +-- single greedy decode  -> "gains" appear (batching)
        |         +-- expansion statistic   -> 0.280 for an untouched model
        |         +-- threshold repair      -> null still non-zero
        |         +-- exact per-problem test-> detects nothing  (CORRECT)
        |
        +--> 3x rank-32 LoRA self-training rounds
        +--> external distillation arm
                  |
                  v
   matched in stream / volume / evaluation
                  |
                  v
   FINDING: distillation helps on rarely-reached problems,
            self-training does not
                  |
                  v
   STRESS TEST: regression says the asymmetry is a
                by-product of distillation's larger gain
                  |
                  v
   HARM: self-training corrupts baseline-solved problems
         above the measured floor

Think of it as a bathroom scale you calibrated against a known weight. You want to detect a one-kilogram change in yourself. Before trusting any reading, you put a certified ten-kilogram plate on the scale — a quantity that is not supposed to change — and watch. If the number drifts while the plate sits there, the drift is the instrument, not you. The seven failures are seven different ways the scale drifted under a weight that never moved: one from reading the display too quickly (single greedy decode), one from a formula that reports growth for a constant weight (expansion statistic), one from setting the zero-point using the drifting reading itself (threshold on a non-zero null). What survives is a procedure that reads zero for the plate.

Key Concepts

• The measured null: a statistic’s expected value under a condition where nothing actually changed, established by running a frozen control through the identical pipeline. The paper’s central claim is that a null must be measured separately for every statistic reported, because each has its own artifact profile — a statistic is not interpretable merely because a plausible-looking baseline exists.
• Acquisition versus sharpening: the distinction the expansion statistic tries to draw. Acquiring a capability means the model can now solve problems it previously could not reach at all; sharpening means it got more reliable at problems it could already sometimes solve. These call for different interpretations of “self-improvement”, which is why conflating them matters.
• False-discovery-rate control at the per-problem level: raising the unit of analysis from the benchmark mean to the individual problem multiplies the number of comparisons, so any per-problem claim needs multiple-testing control. The paper’s exact test holds its behaviour constant across the choice of rule, error rate and pool size, which is what makes it reusable rather than tuned.

Framework Shift

Before (mean accuracy, no null):
  train -> eval once -> diff vs baseline -> report gain
  Noise source: unbounded, unmeasured, indistinguishable
                from signal

After (transition-level accounting with a null):
  +--- frozen control ---+     +--- trained arms ---+
  | identical pipeline   |     | identical pipeline |
  | expected: no change  |     | question: change?  |
  +----------+-----------+     +---------+----------+
             |                           |
             +-----------+---------------+
                         v
          compare each statistic against its measured null
          per-problem exact test, FDR-controlled
                         v
          report only what survives the floor

From trusting a difference because it looks large, to trusting it only after watching the same instrument report nothing on something that provably did not change, the core shift is paying for a control before paying for a claim.

Expert Assessment

Problem choice: Outstanding, and overdue. The transition-level metric was adopted because it is more informative, and it genuinely is — but a more sensitive instrument without a noise floor is a regression in scientific quality, not an improvement. Framing the paper as an audit rather than a new method is the right call, because the field’s problem here is methodological, not architectural.

Method maturity: The cleverness is in the negative result. Replicating rank-32 LoRA self-training on one 8B model is cheap; the insight is that you can get a null for free from replicates the study already owns, and that most published designs do not possess enough replicates to build one. That second half is the uncomfortable part, and the paper states it plainly instead of softening it.

Experimental integrity: This is the paper’s whole subject, and it holds itself to the standard it demands. Reporting that the popular threshold repair fails to replicate, that the expansion statistic scores an untrained model at 0.280, and that the headline asymmetry is rejected by its own regression (p under 10^-8) is exactly the behaviour the audit is arguing for. The scope is honestly narrow — three rounds, one model, one rank — and the authors do not overclaim beyond it.

Writing quality: The seven failures are the paper’s substance and they are listed as mechanisms rather than as a narrative, which is right for a methods paper. The one place it would gain from rewriting is the ladder of arms: the stream/volume/evaluation matching deserves more explicit treatment, because that matching is what legitimises comparing distillation against self-training at all.

Verdict: strong accept — a necessary corrective that makes a popular class of result falsifiable, and hands over an exact test that reports nothing when nothing happened.

Takeaways

• Before reporting any per-problem gain or loss statistic, run a frozen control through the same pipeline. If the statistic moves for a model that never trained, the statistic is measuring your harness.
• Treat each reported statistic as needing its own null. Sharing one baseline across a family of metrics is how a non-zero null survives into a conclusion.
• Beware repairs that calibrate a threshold on a null known to be non-zero. That does not remove the artifact; it encodes it.
• Self-training’s cost is visible only at transition level: it corrupts problems that were solved at baseline at rates above the measured floor. A flat mean accuracy would have reported this as no change at all.