What FID Hides: A Scalar Cannot Rank, Test and Diagnose at Once

Paper: 2608.24881 Authors: Hao Chen Categories: stat.ML, cs.LG

The Gap

Generative models are ranked by FID and KID. That is not a methodological preference; it is the field’s default, and it carries three distinct jobs at once — ranking models by how far they depart from real data, testing whether a departure is statistically real, and diagnosing what kind of departure it is. A single scalar is asked to do all three, and the paper shows it cannot do any of them reliably.

The failure has a concrete demonstration rather than an argument. FID summarises a distribution by its first two moments. So on ImageNet, images optimized only to match the reference Inception mean and covariance obtain FID 24.7, versus 58.6 for held-out real images — lower being better. The optimized images are visually unrecognizable and they beat real data on the metric. This is not FID being misused; it is FID reporting exactly what it measures. The moment restriction is the metric.

The second failure is subtler and more damaging for practice. FID and KID are scalar discrepancies, unchanged when the two samples are exchanged. A number that does not change under swapping cannot encode the direction of a dispersion change — and the direction is precisely what you need to distinguish mode collapse (under-dispersion) from over-dispersion. Two opposite pathologies produce the same statistic.

The third failure is statistical. A reported scalar gap is not a calibrated test against sampling variation. Without a null distribution, “model A beats model B by 3 FID” is an observation, not a finding.

   ONE SCALAR, THREE INCOMPATIBLE JOBS

   [ RANK ]  which model is further from real?   -> a score
   [ TEST ]  is the gap beyond sampling noise?   -> a p-value
   [ DIAGNOSE ] what KIND of departure?          -> a direction
                    |
        all three currently answered by FID / KID
                    |
                    v
   [1] MOMENT RESTRICTION
       ImageNet: images matching only the reference
       mean+covariance  ->  FID 24.7
       held-out REAL images ->  FID 58.6   (lower is better)
       -> unrecognizable images beat real data

   [2] SYMMETRY
       FID/KID unchanged when the two samples are SWAPPED
       -> cannot encode direction
       -> under-dispersion (mode collapse) vs over-dispersion
          produce the same number

   [3] NO NULL
       a scalar gap is not calibrated against sampling variation

The Increment

One sentence: Before this paper, one number was asked to rank, test and diagnose; after it, ZID reports three linked outputs — a magnitude index, a permutation p-value, and a signed dispersion readout — and detects departures on which FID is flat or reversed.

Core Mechanism

The diagnosis drives the design, so ZID is assembled to remove each specific restriction rather than to improve a scalar.

Six standardized arms across two kernel families. ZID combines six location- and dispersion-sensitive arms drawn from a rank graph (RISE) and Gaussian kernels (GPK at two bandwidths). The multiplicity is the point: a single kernel at a single bandwidth has its own blind spots, and taking an aggregate over location- and dispersion-sensitive statistics across families is how the metric stops depending on one summary. Standardizing the arms is what makes them combinable at all.

Three linked outputs instead of one overloaded scalar. This is the structural fix, and it is the right shape for the problem:

• An index for ranking departure magnitude — the job FID was doing, now measured without the moment restriction.
• A permutation p-value for testing distributional equality — supplying the null that a bare scalar gap lacks, so a gap can be distinguished from sampling variation.
• A signed dispersion readout for diagnosis — the direction, which a swap-invariant statistic structurally cannot provide.

Three outputs for three incompatible jobs, with no attempt to compress them back into one number.

The experiments target exactly the cases the diagnosis predicts will break FID. In controlled sweeps, ZID detects a broad range of departures, and its score tracks increasing severity along the corresponding sweeps, including cases in which FID is flat or reversed. Monotone tracking of severity is the property you want from a ranking index and it is the property a moment-based scalar fails to provide. On DiT-XL/2 and SiT-XL/2 guidance sweeps, ZID detects departure from real data, and the signed readout labels the high-guidance diversity collapse as under-dispersion — the diagnosis that a swap-invariant statistic cannot express.

   ZID: THREE OUTPUTS FOR THREE JOBS

   inputs: generated sample, real sample
        |
        v
   SIX STANDARDIZED ARMS
     rank graph (RISE)          } location- and
     Gaussian kernels (GPK x2)  } dispersion-sensitive
        |
        +-----------------+------------------+
        v                 v                  v
   [ INDEX ]        [ PERMUTATION ]     [ SIGNED DISPERSION ]
   rank departure   p-value: are the    direction: under- vs
   magnitude        distributions       over-dispersion
                    equal?
   (replaces the    (adds the null      (impossible for a
    FID ranking)     FID lacks)          swap-invariant scalar)
        |                 |                  |
        +-----------------+------------------+
                          v
   on DiT-XL/2, SiT-XL/2 guidance sweeps:
     detects departure AND labels the
     high-guidance collapse as UNDER-dispersion

Think of it as replacing a single bathroom scale with a full check-up. Stepping on a scale gives one number, and it is genuinely useful for one question — roughly, how heavy am I. But if you ask that same number whether you are actually ill, and in which direction, it cannot answer, and it will happily return a reassuring figure for someone with a serious problem. A check-up separates the outputs instead: a measurement you can track over time, a test against a reference range that tells you whether a difference is real or noise, and a signed reading that distinguishes “too much of X” from “too little of X”. The paper’s ImageNet example is the scale reporting a great number for an unrecognizable image — not a broken scale, but a scale answering a question it was never able to answer.

Key Concepts

• The moment restriction as the root cause: FID summarises a distribution by mean and covariance. Its famous failure on optimized images is not a misuse but a direct consequence — anything matching those two moments scores well, regardless of what it looks like. Naming this as the mechanism rather than as an anecdote is what motivates replacing the summary.
• Swap invariance and the missing direction: FID and KID are unchanged when the two samples are exchanged. This is a structural property, and it makes the metric incapable of distinguishing mode collapse from over-dispersion — two opposite problems reported identically.
• Calibration against sampling variation: a p-value from a permutation test. Without it, a scalar gap cannot be separated from the noise of finite sampling, which means ranking and testing are not the same activity and should not share an output.
• Severity tracking: the property that a ranking index increases monotonically as the departure worsens. It is what makes a metric usable for model selection, and it is what FID fails to deliver in the paper’s sweeps.

Framework Shift

Before (one scalar, three jobs):
  FID / KID -> a number
    rank:     yes, under a moment restriction
    test:     no null distribution
    diagnose: impossible (swap-invariant)
  -> unrecognizable images can beat real data
  -> mode collapse and over-dispersion look identical

After (three outputs, matched to jobs):
  ZID
    index        -> rank departure magnitude
    permutation  -> p-value against sampling noise
    signed read  -> under- vs over-dispersion
  -> tracks severity where FID is flat or reversed
  -> labels high-guidance collapse as under-dispersion

From asking one number to rank, test and diagnose at once, to separating outputs by the question they answer, the core shift is recognising that a swap-invariant scalar cannot express direction, and a moment summary cannot express severity.

Expert Assessment

Problem choice: Excellent, and it has been needed for years. FID is one of the most consequential numbers in generative modelling — it decides which papers look better — and the community has long known it was flawed without a clean account of which specific jobs it fails, or what to replace them with.

Method maturity: The design follows the diagnosis rather than decorating it, which is the right way to build a metric. Six standardized arms across a rank graph and two kernel bandwidths is a reasonable construction for breadth without an unbounded hyperparameter search, and the three-output structure refuses the temptation to squeeze everything back into one number. The open question is sensitivity to the arm set: an aggregate over six statistics needs a defensible weighting, and a reader will want to know how much the index depends on that choice.

Experimental integrity: The ImageNet demonstration is the strongest kind of evidence — a construction that exposes what the metric actually measures, rather than an argument about what it might. Reporting cases where FID is flat or reversed is candid and important, since those are precisely the situations that would have invalidated the method had they gone unmentioned. The evaluation is largely in controlled sweeps and two DiT/SiT families, which is appropriate for a metric paper, though broader adoption will depend on how it behaves on the noisier, less controlled comparisons people actually run.

Writing quality: The framing of three incompatible jobs answered by one scalar is stated immediately and organises everything that follows. The paper would benefit from one concrete worked comparison — a pair of models where FID and ZID disagree, with the images — since that is the evidence that would convince a practitioner to switch.

Verdict: strong accept — it replaces an overloaded metric with outputs matched to the questions being asked, and it demonstrates the failure mode with a construction rather than an argument.

Takeaways

• Stop using one scalar for ranking, significance testing and diagnosis. These are three questions and they need three outputs; compressing them loses at least one.
• Check swap invariance before trusting a distance metric to describe a direction. If exchanging the samples does not change the number, the metric cannot express which way things moved.
• Read the moment restriction in any moment-based metric. FID rewards matching mean and covariance, so optimization pressure against it will find images that match those two things and nothing else.
• When reporting a metric gap, report a null alongside it. A scalar difference without a calibrated test cannot be distinguished from sampling noise, no matter how many decimals it has.