Progressive Seed Pruning: Spend Your Inference Budget Where It Matters

Paper: 2607.21591 Authors: Rogerio Guimaraes, Pietro Perona Categories: cs.CV

The Gap

Diffusion and flow-matching models are the workhorses of image generation, but their inference-time scaling story is embarrassingly thin compared to what autoregressive LLMs enjoy. The core problem: final image quality is extremely sensitive to the initial noise seed. Change the random seed, get a wildly different result — some great, some mediocre.

Existing approaches handle this in two ways. Best-of-N simply denoises N seeds completely and picks the highest-scoring one — wasteful, since you burn full compute on seeds that were obviously bad at step 2. Tree search / importance sampling methods maintain a fixed number of candidates at every timestep, spreading compute uniformly across the trajectory. The shared blind spot: they all maintain a constant memory footprint throughout inference, never asking whether some seeds deserve more attention than others.

The authors’ insight is deceptively simple: you can predict final quality from intermediate denoised images. If a seed looks bad at 25% denoising, it’s not going to magically become great at 100%. So why keep evaluating it? Relax the constant-memory constraint, front-load your exploration, and prune hard.

[Seed sensitivity: quality varies wildly across random seeds]
                            |
                            v
[Prior constant-memory methods spread compute uniformly]
                            |
                            v
[Key insight: intermediate scores predict final quality]
                            |
                            v
[PSP: explore many seeds early, prune progressively]
                            |
                            v
[Same total model evaluations, better final winners]

The Increment

One sentence: Before this paper, inference-time scaling for diffusion models meant picking the best from a fixed set; after this paper, it means a dynamic tournament where early losers get eliminated and survivors receive concentrated compute.

Core Mechanism

Progressive Seed Pruning works in stages. You start with a large pool of N random seeds (say, 64). Instead of denoising all of them to completion, you partially denoise every seed to an intermediate timestep — maybe 25% of the way. At this checkpoint, you score each candidate using a reward model (automated metrics like prompt-alignment scores, or any differentiable/black-box reward). The bottom fraction gets pruned. You repeat: take the survivors, denoise them further to 50%, score again, prune again. By the final stage, you might have just 1–4 candidates completing full denoising.

The critical constraint is that the total number of model evaluations is fixed. PSP doesn’t add compute — it redistributes it. You spend many cheap, partial evaluations early (where individual steps are fast) and concentrate expensive full evaluations only on the seeds that earned it. This is possible because denoising from timestep 0 to 25% uses fewer resources than denoising from 75% to 100%, so the math works out: exploring broadly early is cheap.

The scoring function is the linchpin. It needs to be predictive of final quality from partial denoising. The paper uses reward models and automated metrics (GenEval), but the framework is agnostic to the specific scorer — any signal that correlates with final quality at intermediate steps will do.

N seeds (64)
    |
    v
[Partial Denoise to 25%]
    |
    v
[Score all 64] --> [Prune to 16]
    |
    v
[Partial Denoise to 50%]
    |
    v
[Score all 16] --> [Prune to 4]
    |
    v
[Partial Denoise to 75%]
    |
    v
[Score all 4] --> [Prune to 1]
    |
    v
[Full Denoise] --> Final image

The Audition Metaphor: Imagine you’re casting a movie and you have 1,000 audition tapes, but only budget to watch a total of 1,000 minutes of footage. The old way: watch all 1,000 tapes end-to-end (1 minute each), then rank them. PSP’s way: watch the first 15 seconds of all 1,000 tapes (250 minutes). You can already tell who has screen presence and who doesn’t. Cut to 250 actors. Now watch 1 minute of each (another 250 minutes). Cut to 60. Watch 2 minutes each (120 minutes). Cut to 15. Watch the full tape (375 minutes). You’ve spent roughly the same total time, but your finalists got 25× more screen time than your early cuts — and your winner is much better than if you’d given everyone exactly 1 minute.

The mapping is load-bearing: audition tapes are noise seeds, watching footage is model evaluations, cutting actors is pruning, and the fixed budget is total compute. The core bet — that you can spot talent in 15 seconds — maps directly to the paper’s assumption that intermediate quality predicts final quality.

Key Concepts

• Seed sensitivity: Every diffusion model starts from random noise. The same prompt + different noise = completely different images, ranging from stunning to incoherent. This isn’t a bug; it’s the nature of the sampling process. The practical consequence is that single-shot generation is a lottery. If you care about quality, you need to generate multiple candidates — but how you spend that extra compute matters enormously.

• Inference-time scaling: At training time, you’ve already baked in the model’s capabilities. At inference time, the question is: given a fixed compute budget, how do you extract the best possible output? For LLMs, this is well-studied (chain-of-thought, best-of-N, tree search). For diffusion models, the field has mostly relied on brute-force best-of-N. PSP shows there’s a smarter allocation strategy — think of it as the difference between giving every student the same textbook vs. tutoring the promising ones more intensively.

• Progressive pruning as compute allocation: The deeper principle is that not all compute is equal. Evaluating a nearly-finished image is expensive (you’ve already done 90% of the denoising steps). Evaluating a barely-started image is cheap. So early-stage evaluations have a better compute-to-information ratio. PSP exploits this asymmetry: cast a wide net when evaluations are cheap, narrow the field when they get expensive. It’s the same logic behind multi-stage hiring processes, drug trial phases, or venture capital funnels.

Framework Shift

Before (mainstream):                   After (this paper):

seed1 ---------------->[full denoise]  seed1 --+
seed2 ---------------->[full denoise]  seed2 --+--[denoise 25%]
seed3 ---------------->[full denoise]  seed3 --+       |
  ...                                  seed4 --+   [score+prune]
seedN ---------------->[full denoise]  seed5 --+       |
      |                                      |    [denoise 50%]
  [pick best]                               ...       |
                                                  [score+prune]
                                              survivors |
                                                  [denoise 75%]
                                                      ...
                                                  [winner]

Equal work per seed, uniform.       Unequal work, earns its keep.

From uniform compute allocation to dynamic compute allocation, the core shift is treating inference as a tournament bracket rather than a parallel race.

Expert Assessment

Problem choice: Real gap, well-positioned. Inference-time scaling for diffusion is genuinely underdeveloped relative to LLMs, and the field has been coasting on best-of-N as the default. The observation that intermediate quality predicts final quality is not new per se — it’s implicit in every paper that uses early stopping or guidance schedules — but nobody had systematically exploited it for seed selection. Good instinct to formalize this.

Method maturity: Cleverer than brute force, but not over-engineered. The core idea is essentially a compute-reallocating tournament, which is conceptually simple. The risk is that the method’s effectiveness is tightly coupled to the quality of the intermediate scoring function — if your reward model is noisy at early timesteps, you might prune good seeds prematurely. The paper would benefit from an ablation on scoring accuracy at different denoising stages. There may be simpler approximations (e.g., just running two stages of pruning instead of many) that capture most of the benefit.

Experimental integrity: The baselines are reasonable — best-of-N, importance sampling, and tree search cover the main competing paradigms. GenEval is standard for automated evaluation, and the addition of human evaluation for prompt-alignment is a plus. The “matched compute” framing is the right way to compare. One concern: how sensitive are the results to the number of pruning stages and the pruning schedule? If the method requires careful hyperparameter tuning per model/reward combination, that limits practical adoption. I’d want to see sensitivity analysis.

Writing quality: The abstract is crisp and well-structured — it clearly states the gap, the insight, and the evidence. The project page and code availability are good signs. The framing around “relaxing the constant-memory constraint” is a nice way to position the contribution. Without seeing the full paper, I’d flag that the related work section is likely where corners could be cut — there’s a rich literature on compute-optimal inference in LLMs that deserves a careful mapping.

Verdict: weak accept — Clean idea, solid evaluation framing, addresses a real gap, but the method’s practical ceiling depends heavily on scoring function quality, which deserves deeper investigation.

Takeaways

Three things worth stealing:

1. The tournament framing for any multi-candidate generation task: If you’re generating multiple candidates and picking the best, ask yourself: can I evaluate intermediate states cheaply? If yes, prune early and reallocate compute. This transfers to video generation, text-to-3D, music synthesis — anywhere you run multiple expensive generation passes.

2. Compute-as-currency thinking: Don’t treat inference budget as “run N models.” Think of it as a budget you allocate dynamically. Early evaluations are cheap and informative; late evaluations are expensive and less discriminative. Front-load your information gathering. This is the same principle behind multi-armed bandit algorithms and Bayesian optimization, just applied to diffusion seeds.

3. Intermediate quality as a proxy: If you’re building a reward model for generative outputs, consider whether it works on partially-generated outputs. A reward model that only scores finished images forces you into best-of-N. One that scores half-finished images unlocks progressive strategies. This is a design constraint worth building toward from the start.