DiffusionBench: Time to Stop Evaluating Diffusion Transformers on ImageNet Alone

Paper: 2606.24888
Authors: Xingjian Leng, Jaskirat Singh, Zhanhao Liang, Ethan Smith, Martin Bell, Aninda Saha, Yuhui Yuan, Liang Zheng
Categories: cs.CV

The Gap

Over the past two years, diffusion transformer (DiT) research has converged to a single evaluation setup: class-conditional generation on ImageNet. Almost every new architecture, sampling method, or training trick reports FID, sFID, and IS on ImageNet-256/512. The unstated assumption is that improvements on ImageNet generalize to the more practical text-to-image (T2I) setting. But T2I is often skipped because it’s perceived as too costly or inconvenient to train and evaluate. This paper challenges that perception head-on. The authors build NanoGen, a unified DiT training framework that makes training T2I models just as cheap as ImageNet—and then show that method rankings on ImageNet have no correlation with rankings on T2I (Pearson between -0.377 and -0.580). The gap is clear: the community is using a single, cheap proxy that doesn’t predict actual progress.

Problem: DiT eval locked to ImageNet (class-conditional)
         |
         v
Assumption: ImageNet FID improvements -> generalizable to T2I
         |
         v
Method: NanoGen (unified training, 12 lines config switch)
         |
         v
Evidence: Train 21 models. ImageNet rank vs T2I rank:
          Pearson = -0.377 to -0.580 (no correlation)
         |
         v
Conclusion: Must evaluate on both. Propose DiffusionBench.

The Increment

One sentence: Before this paper, DiT methods were evaluated solely on ImageNet under the untested assumption that results transfer to T2I; after this paper, we have a unified training framework, evidence that the assumption is false, and a two-axis benchmark that researchers should report instead.

Core Mechanism

NanoGen is not a new model—it’s a training framework that abstracts away the differences between ImageNet and T2I training pipelines. The key idea is that both setups share the same core diffusion loop (forward noise process + denoising objective), and only differ in conditioning: images come with class labels or text captions. NanoGen provides a modular architecture with interchangeable components:

• Encoder: supports RAE, VAE, pixel-space, and MeanFlow diffusion methods (i.e., different ways to map images to latent space or work directly in pixel space).
• Conditioning source: class embeddings (for ImageNet) or text embeddings from a frozen CLIP/T5 encoder (for T2I).
• Training loop: same diffusion noise schedule, same loss function (e.g., L2 on noise prediction or velocity), same optimizer config.

Switching from ImageNet to T2I requires changing exactly 12 lines in the configuration file: replacing class labels with text captions, pointing to a different dataset, and adjusting the batch size / resolution. That’s it. The training cost is comparable (~same GPU-hours for a similar number of steps). This allows fair, apples-to-apples comparison across tasks.

           +---------------------------+
           |     Config file          |
           |  (12 lines change)       |
           +-----------+--------------+
                       |
                       v
   +-------------------+--------------------+
   |          NanoGen Training Pipeline    |
   +-------------------+--------------------+
   |  Encoder          |  Conditioner      |
   |  [RAE|VAE|pixel]  |  [class|text]    |
   +--------+----------+--------+----------+
            |                   |
            v                   v
   +--------+----------+--------+----------+
   |  Forward Diffusion | Loss (noise pred)|
   +-------------------+--------------------+
   |         Output: trained model          |
   +-------------------+--------------------+

Structural metaphor: Think of NanoGen as a universal power drill. The drill body (training loop) is the same. But you can swap out the drill bit (encoder type: RAE, VAE, etc.) and the work surface (conditioning source: ImageNet with class labels, or COCO with text captions). Changing from one task to the other is as simple as attaching a different bit and adjusting the speed dial (12 config lines). The core question the paper asks is: does a bit that drills faster into wood (ImageNet) also drill faster into concrete (T2I)? The answer, experimentally, is no—the optimal bit for wood is different from the optimal bit for concrete. So researchers must test on both surfaces before claiming their bit is universally better.

Key Concepts

• Evaluation correlation (Pearson r): A value between -1 and 1 that measures how strongly two rankings move together. r = 1 means perfect same order; r = 0 means no relationship; r = -1 means opposite order. The paper finds r ≈ -0.5, meaning if you improve ImageNet FID, you’re slightly *more likely to worsen T2I FID. That’s a spectacular failure of the proxy evaluation. For intuition: imagine ranking cars by fuel efficiency on highways only—then discovering that highway ranking is negatively correlated with city ranking. You’d never buy a car based solely on highway numbers.

• Unified framework vs. one-off codebases: Most DiT papers use custom code that differs in training details (learning rate, batch size, noise schedule, etc.). These differences can mask or amplify real improvements. NanoGen fixes all those variables, so any performance difference comes from the method itself. This is the difference between comparing two chefs’ signature dishes (different kitchens, different knives) vs. a cooking competition where everyone uses the same stovetop and pans.

Framework Shift

Before (mainstream approach):          After (this paper):
+----------------------------------+  +------------------------------------+
| Train on ImageNet only           |  | Train on ImageNet + T2I (same cost)|
| Eval: FID, sFID, IS on ImageNet  |  | Eval: DiffusionBench (2 metrics x 2|
| Assume results transfer to T2I   |  | tasks).                       |
+----------------------------------+  +------------------------------------+
| Method A -> FID=2.0 (best)       |  | Method A: ImageNet FID=2.0 (best) |
| Method B -> FID=2.2 (worse)      |  |          T2I FID=15.0 (worst)    |
+----------------------------------+  | Method B: ImageNet FID=2.2 (worse)|
                                      |          T2I FID=13.0 (best)     |
                                      +------------------------------------+
                                      | Winner: unclear without both axes.|
                                      +------------------------------------+

One sentence: From evaluating on a single synthetic proxy (ImageNet) to evaluating on two complementary tasks (class-conditional + text-conditional), the core shift is recognizing that different conditioning spaces stress different aspects of generative capability.

Expert Assessment

Problem choice: Real and urgent. The community has been drifting toward a single metric that is cheap but misleading. This paper callously dismantles the pillar of convenience and forces a rethinking. It sits exactly at the inflection point where DiT papers are piling up without clear practical guidance.

Method maturity: Clever insight, not brute force. NanoGen itself is straightforward engineering—a unified config system—but the experimental design (train 21 models, compute correlation) is elegant and sufficient. Simpler approaches (e.g., just comparing two or three methods on T2I) would have been too weak. The 12-line switch is a nice engineering hook.

Experimental integrity: Baselines are fair—they control for everything except the method. The numbers are internally consistent. One red flag: the paper only uses one T2I dataset (likely COCO? Abstract doesn’t specify but typical). Generalization to other T2I benchmarks (e.g., DrawBench, PartiPrompts) is not tested. Also, FID on T2I is notoriously unstable; they should have included an additional metric like CLIP score. However, the correlation trend is strong enough that even with noise, the conclusion holds.

Writing quality: The abstract is punchy, the intro builds the gap well. The “12 lines of config change” hook is memorable. The methodology section could be more precise about the exact correlation numbers and their confidence intervals. If I had to rewrite one section, it would be the Discussion—they should speculate *why the correlation is negative. Is ImageNet favoring models that overfit to limited classes? Is the text conditioning fundamentally different? Lacking that analysis makes the paper feel incomplete.

Verdict: strong accept — This is an important wake-up call backed by clean evidence and a practical tool. It reshapes how the community should evaluate DiT research going forward.

Takeaways

1. Never trust a single proxy task. If you’re evaluating a generative model, test it on at least two conditioning distributions. This paper shows that the correlation can even be negative. Practitioners should build multi-task evaluation suites into their training pipelines from day one.

2. Unified training frameworks are worth building. NanoGen’s approach of abstracting the training loop and making task switching a config change is a pattern that can transfer to other domains (e.g., video generation, audio generation). Steal the idea of a modular “conditioner + encoder + diffusion core” architecture.

3. Correlation analysis as a meta-tool. The technique of comparing method rankings across tasks can be applied anywhere you suspect a proxy metric is misleading (e.g., image classification accuracy vs. robustness). The paper’s simple Pearson calculation on 21 points is a template you can copy for your own domain.