On-the-fly Repulsion in the Contextual Space for Rich Diversity in Diffusion Transformers

Paper: 2603.28762 Authors: Omer Dahary, Benaya Koren, Daniel Garibi, Daniel Cohen-Or Categories: cs.CV, cs.AI, cs.GR, cs.LG

The Gap

Modern text-to-image diffusion models like DALL-E and Stable Diffusion have mastered semantic alignment—they understand prompts well. But they suffer from “typicality bias”: given “a cat on a sofa,” they generate the same boring tabby on the same beige couch, over and over. Creative applications need variety, not just correctness.

Prior approaches hit a fundamental trade-off. Input-space methods (modifying prompts or noise) require expensive iterative optimization to incorporate feedback from the generation process. Latent-space methods (pushing apart intermediate images) act too late—the visual structure is already committed, so repulsion creates artifacts and breaks coherence.

Problem: Typicality bias in T2I models
   |
   v
Existing approaches:
   |
   +---> Input-space diversity -----> Costly optimization loop
   |                                   (slow, impractical)
   |
   +---> Latent-space repulsion ----> Acts on committed structure
                                       (artifacts, broken coherence)
   |
   v
Gap: Need intervention point that's
     - Structurally informed (not blind)
     - Compositionally flexible (not fixed)
     - Computationally cheap (not iterative)
   |
   v
This paper: Contextual Space repulsion
   - Intervene in attention channels
   - Between blocks where text meets emerging structure
   - On-the-fly, single forward pass
   |
   v
Evidence: Rich diversity + visual fidelity
          Works on Turbo/distilled models
   |
   v
Conclusion: Attention space is the sweet spot
            for diversity intervention

The Increment

One sentence: Before this paper, diversity required choosing between expensive optimization or visual artifacts; after, you can inject rich variation in a single forward pass by repelling in the attention space where text and image structure meet.

Core Mechanism

The method operates on Diffusion Transformers during generation. At each denoising step, the model processes noisy latents through transformer blocks. Each block contains multimodal attention layers where text tokens interact with image patch tokens. This is where text conditioning gets enriched with emerging visual structure.

The intervention happens between transformer blocks. After a block computes attention but before the next block processes it, the method applies repulsion in the contextual space—the high-dimensional space where text and image representations are jointly embedded. Specifically, it identifies attention features that are becoming too similar across multiple generation samples and pushes them apart.

The repulsion is computed on-the-fly during the forward pass. For each batch of images being generated, the method compares their contextual representations (the attention outputs) and applies a repulsive force proportional to their similarity. This force redirects the attention trajectory without requiring backpropagation or iterative refinement.

Generation batch (N images from same prompt):
   |
   v
Transformer Block i:
   [Text tokens] + [Image patches] ---> Multimodal Attention
                                         |
                                         v
                                    Attention outputs
                                    (contextual features)
   |
   v
Repulsion injection point:
   
   Sample 1: [f1] ----\
   Sample 2: [f2] -----+---> Compute pairwise similarity
   Sample 3: [f3] ----/      |
                             v
                        Apply repulsive force:
                        f'_i = f_i + alpha * sum_j(repel(f_i, f_j))
                             |
                             v
                        Push apart similar features
   |
   v
Transformer Block i+1:
   Receives diversified contextual features
   Continues generation with redirected trajectory

Think of it like a group of hikers following the same trail description. Input-space methods are like giving each hiker a slightly different map before they start—but you need to simulate their entire journey to know if the maps lead to different destinations, then iterate. Latent-space methods are like physically pushing hikers apart when they’re already halfway up the mountain—they stumble and fall.

This paper’s approach is like having a guide who walks alongside the group. At key decision points (trail forks), the guide notices when hikers are clustering together and gently suggests different paths to each person. The guide acts at moments when the trail structure is visible (they can see the fork) but the choice isn’t yet committed (they haven’t taken the turn). The hikers naturally diverge without needing to backtrack or stumble.

The “contextual space” is that moment at the trail fork where both the map (text) and the terrain (emerging image structure) are visible. The “repulsion” is the guide’s gentle nudge. The “on-the-fly” means it happens during the hike itself, not through pre-planning or post-correction.

Key Concepts

• Contextual Space: In a Diffusion Transformer, text and image don’t live in separate worlds. The multimodal attention layers create a shared representation space where text tokens and image patch tokens interact. This is the contextual space—it’s not the raw text embedding, not the pixel-level latent, but the intermediate representation where semantic meaning (from text) is being actively woven into spatial structure (from image patches). Imagine a conversation where two people are building a shared mental model—the contextual space is that shared model, not either person’s private thoughts. When you apply repulsion here, you’re diversifying the shared understanding before it crystallizes into concrete visual decisions.

• Typicality Bias: Diffusion models are trained on internet-scale data where certain visual patterns dominate. “Cat on sofa” appears millions of times with similar compositions—tabby cats, beige sofas, centered framing. The model learns these statistical modes and gravitates toward them. It’s not a bug; it’s overfitting to the training distribution’s peaks. The model is like a chef who’s cooked the same popular dish so many times that even when asked to “make something with chicken,” they default to the bestseller. Typicality bias is the model’s muscle memory for common patterns.

• On-the-fly Intervention: Traditional diversity methods require iterative optimization—generate samples, measure diversity, adjust inputs, regenerate. This paper’s method injects diversity during a single forward pass. “On-the-fly” means the repulsion force is computed and applied in real-time as the transformer processes each block, without stopping to backpropagate or re-plan. It’s like adjusting your steering while driving, not stopping to recalculate your route. This is crucial for efficiency and enables the method to work on fast “Turbo” models where there’s no time for iteration.

Framework Shift

Before (mainstream approach):        After (this paper):

Input Space Intervention:            Contextual Space Intervention:

[Prompt] --modify--> [Prompt']       [Prompt] -----> [Attention]
    |                    |                              |  ^
    v                    v                              |  |
[Generate] -----> [Evaluate]         [Text] + [Image] -+  |
    |                    |               structure         |
    v                    |                  |              |
[Diverse?] --no--> [Iterate]               v              |
    |                                   [Repel] -----> [Redirect]
    yes                                     |
    |                                       v
    v                                   [Continue]
[Done] (slow)                               |
                                            v
OR:                                     [Done] (fast)

Latent Space Intervention:           Key difference:
                                     - Acts at decision points
[Generate] -----> [Latent]           - Structurally informed
    |                |                - Single forward pass
    v                v
[Committed] --> [Push apart]
    |                |
    v                v
[Artifacts]      [Broken]

From iterative optimization or post-hoc correction to real-time trajectory redirection, the core shift is intervening where structure is informed but not yet committed.

Expert Assessment

Problem choice: This is a real gap. Typicality bias is a well-documented pain point in creative applications of diffusion models. The problem sits at the intersection of model behavior and user needs—technically interesting and practically relevant. It’s not manufactured; practitioners genuinely struggle with getting diverse outputs without quality degradation.

Method maturity: The insight is clever—attention space as the intervention point is non-obvious and well-motivated. However, the paper doesn’t deeply explore why this specific location works beyond intuition. The repulsion mechanism itself is relatively straightforward (pairwise similarity-based pushing), not a novel algorithmic contribution. A simpler approach might be stochastic attention dropout, but the authors don’t compare against such lightweight baselines.

Experimental integrity: The baselines are reasonable (input noise perturbation, latent-space repulsion), but the paper lacks comparison with recent diversity-focused methods like DivCon or semantic guidance variants. The metrics (diversity scores, FID, CLIP similarity) are standard but don’t capture subjective creative value—what looks “richly diverse” to a designer? The Turbo model results are impressive and suggest robustness, but the paper doesn’t show failure cases or discuss when the method breaks down.

Writing quality: The introduction and motivation are strong. The method section is clear but could benefit from ablation studies on where exactly to inject repulsion (which blocks? which layers?). The related work section glosses over recent attention-manipulation techniques in diffusion models. The conclusion oversells the generality—“rich diversity” is demonstrated on specific prompt types, not systematically across domains.

Verdict: weak accept — The method addresses a real problem with a practical solution that works on modern fast models, but the contribution is more engineering insight than fundamental advance, and the evaluation could be more rigorous.

Takeaways

Practitioners can steal the core idea: intervene in attention layers, not input or output spaces. This applies beyond diffusion models—any transformer-based generative model (language, audio, video) has attention layers where you can inject diversity without breaking coherence. The specific technique: compute pairwise similarity of attention outputs across a batch, apply repulsive forces proportional to similarity, do it between blocks where conditioning meets structure.

For implementation: the paper suggests a small computational overhead (they claim ~10% slowdown), which is practical. The key is identifying the right intervention point—not too early (before structure emerges) or too late (after commitment). In your domain, look for the layer where semantic input and structural output first interact.

A transferable framing: diversity interventions should be “trajectory redirections,” not “endpoint corrections.” Act at decision points, not after decisions are made.