RepFusion: Let Multimodal LLMs Denoise Visual Representations Directly

Paper: 2606.14700
Authors: Xichen Pan, Aashu Singh, Satya Narayan Shukla, Xiangjun Fan, Shlok Kumar Mishra, Saining Xie
Categories: cs.CV

The Gap

The text-to-image pipeline has a weird asymmetry. We use a massive pretrained LLM (or T5, CLIP, etc.) to encode the text prompt, but the image generation — the actual denoising — is handled by a separately trained UNet or DiT that starts from scratch. Then along came Representation Autoencoders (RAEs) like MaskDiT or REPA, which showed you don’t need to generate raw pixels; you can generate semantically structured visual representations (e.g., DINOv2 features) instead. This creates a latent space much closer to LLM-compatible semantics. Good.

But the denoiser itself is still newly trained. Even if the target representations are LLM-friendly, the process of going from noise to those targets remains a heavyweight learned module. Why not use the LLM (or rather, an MLLM that can ingest images) directly as part of the denoising process? Multimodal LLMs already have MLP projectors that align clean visual features with the language model. The key insight: that projector also works on noisy visual features, because the MLLM has seen enough corrupted inputs during pretraining to generalize. So you can treat the MLLM itself as a noisy representation encoder, feed it the evolving noisy representation step by step, and use its output as a conditioning signal for a lightweight diffusion transformer. That’s the gap — nobody had tried to repurpose the MLLM as an active denoising component rather than just a frozen text encoders.

[ASCII logic topology: Problem -> Assumption -> Method -> Evidence -> Conclusion]

    Problem: T2I denoisers are trained from scratch, ignoring MLLM priors
         |
         v
    Assumption: MLLM's visual encoder can handle noisy inputs *
    *  (supported by MLLM pretraining data diversity)
         |
         v
    Method: RepFusion - use MLLM as noisy representation encoder,
    output used as DiT conditioning
         |
         v
    Evidence: Controlled comparisons at same inference budget
    (e.g., 400M param DiT) -> RepFusion beats baselines
         |
         v
    Conclusion: MLLMs provide strong denoising priors; test-time
    compute can be spent on repeated MLLM conditioning

The Increment

One sentence: Before RepFusion, the MLLM was a passive text encoder in T2I; after RepFusion, the MLLM becomes an active, iteratively conditioned denoising component, improving quality without increasing denoiser size.

Core Mechanism

RepFusion has three main components: a frozen multimodal LLM (e.g., LLaVA-1.5 with Vicuna-7B), a frozen target representation encoder (e.g., DINOv2), and a small diffusion transformer (DiT, ~400M parameters). The process works like this:

1. Forward process: Start with a target image. Encode it into a clean visual representation using DINOv2. Then add noise to get a noisy representation at timestep t.
2. MLLM conditioning: Feed the noisy representation (after a learned lightweight projection) into the frozen MLLM. The MLLM processes it alongside the text prompt, and the final hidden state (specifically the [CLS] token of the visual branch after the projector) becomes the conditioning signal. This is done at *each denoising step, reusing the MLLM forward pass.
3. Denoising in representation space: The DiT takes the noisy representation and the MLLM conditioning, and predicts the clean representation.
4. Target decoding: After getting the predicted clean representation, decode it to the final image via a lightweight decoder trained on top of the representation (e.g., a small GAN or VAE).

The crucial data flow: noisy representation → MLLM → conditioning → DiT → clean representation. The MLLM is reused at every denoising step, which is expensive but, as the paper shows, productive — the MLLM’s prior adapts to the current noise level.

[ASCII diagram of method internals]

     +---------------------------+
     |   Frozen MLLM (LLaVA)     |
     |  (with MLP projector)     |
     +-----^-----------------^---+
           |                 |
     noisy rep + text    conditioning
           |                 |
           v                 v
     +-----------+    +-------------+
     | DiT denois|--> | Predicted   |
     | (400M par)|    | clean rep   |
     +-----------+    +------^------+
                              |
                    target encoder (DINOv2)

    At each timestep t:
      1. Add noise to representation z_t = sqrt(alpha_t)*z_0 + sqrt(1-alpha_t)*eps
      2. z_t -> project -> MLLM -> conditioning c_t
      3. DiT takes (z_t, c_t) -> predicts z_{t-1}

Structural metaphor: Think of the MLLM as a skilled chef who can taste a half-cooked dish and correct it. The standard T2I pipeline is like having a recipe writer (LLM for text) and a separate apprentice cook (denoiser) who’s never tasted anything before. The apprentice reads the recipe and tries to cook, but makes many mistakes. RepFusion says: let the chef taste the partially cooked dish at every stage and give real-time corrections. The chef’s palate is the MLLM, trained on millions of dishes (images) and recipes (text). The apprentice is the small DiT, which doesn’t need to know cooking theory — it just follows the chef’s corrections. The chef’s feedback (conditioning) is a short verbal note: “more salt, less heat.” Each denoising step, the apprentice hears a new note because the dish has changed. That’s why repeated MLLM conditioning helps: the chef adapts to the current state.

Key Concepts

• Representation Autoencoder (RAE): Instead of generating pixels directly, an RAE learns to map images into a semantic representation space (e.g., DINOv2 features) and then decode back. The denoising happens in that semantic space, which is easier because the representation is more abstract and compact. RepFusion inherits this philosophy but changes who does the denoising.
• MLLM as a noisy processor: Multimodal LLMs (e.g., LLaVA) are typically used on clean images. But during their pretraining, they see many corrupted or noisy images — blurry web images, low-res thumbnails, etc. So their visual encoder learns to extract *robust features. RepFusion exploits this: the MLLM’s projector can align noisy representations with the language model’s space, just as it does for clean ones. The key empirical finding is that the alignment quality degrades gracefully with noise, so the conditioning remains useful at all t.
• Test-time compute scaling: In standard diffusion, the denoiser is reused but its computation per step is constant. Here, the MLLM adds extra compute per step. The paper shows that this extra compute is *more efficient than scaling the denoiser itself — spending compute on the MLLM yields higher FID gains than adding the same number of FLOPs to the DiT.

Framework Shift

Before (mainstream approach):        After (this paper):

   Text Prompt                          Text Prompt
       |                                    |
       v                                    v
   Frozen LLM (text only)              Frozen MLLM (visual+text)
       |                                    |
       v                                    v
   Learned embedding (just text)        Noisy representation (visual)
       |                                    |
       v                                    v
   Large Denoiser (UNet/DiT             Small DiT conditioned by
   trained from scratch)                 MLLM output at each step
       |                                    |
       v                                    v
   Pixel space decoder                  Rep. decoder (small)

   * LLM contributes once             * MLLM contributes at every step
   * No visual feedback               * Visual feedback loop through MLLM
   * Big denoiser bottleneck          * Compute shifted to MLLM reuse

One sentence: From a single-pass text encoder + big scratch denoiser to a recurrent visual-linguistic conditioning loop that leverages pre-existing MLLM capabilities, the core shift is treating the MLLM as a dynamic denoising oracle rather than a static text encoder.

Expert Assessment

Problem choice: Real gap, well-motivated. The field has been obsessed with scaling denoisers (DiT-XL, etc.) while ignoring that we already have powerful multimodal models sitting idle during generation. This paper identifies a concrete inefficiency and exploits it. Positioned at the intersection of T2I and representation learning — timely.

Method maturity: Clever insight, not brute force. The approach is surprisingly simple: just feed noisy representations into a frozen MLLM and use its output as conditioning. No custom architecture or heavy training — only a lightweight MLP to project noisy representations into the MLLM’s input format, and a standard DiT. The paper’s contribution is the *repurposing idea, not new building blocks. That’s high leverage.

Experimental integrity: Fair baselines. They compare against REPA (same DiT architecture, same representation target), LaViT, and standard DiT, all at comparable inference FLOPs by scaling down the DiT when MLLM is used. The metrics (FID, CLIP score, GenEval) hold up. One red flag: they use a relatively old MLLM (LLaVA-1.5). Would results improve with GPT-4V-level MLLMs? Likely, but not tested. Also, the decoder from representation to image is not well-ablated — could the quality be limited by the decoder rather than the denoising?

Writing quality: Well-structured overall, but the “repeated MLLM conditioning” insight is buried in Section 4.2. The abstract promises “test-time compute can be productively spent” but the experiments on this point are scattered. If they consolidated a dedicated table showing FID vs. total FLOPs for different repetitions of MLLM vs. bigger DiT, the paper would be much stronger. Also, Figure 3 (qualitative) is too small; hard to see differences.

Verdict: strong accept — A clean, surprising result that changes how we think about MLLM usage in generation, with practical implications for efficient T2I.

Takeaways

1. Reuse, don’t retrain: If you have a large pretrained multimodal model, don’t just use it for text encoding — feed it the noisy latent at each diffusion step. The overhead is tolerable and yields better quality. This principle applies beyond T2I: any conditional generation task with iterative refinement could benefit from a strong frozen oracle.
2. Test-time compute allocation matters: The paper shows that spending extra FLOPs on RER (repeated MLLM evaluation) is more FID-efficient than spending the same FLOPs on a larger denoiser. Practitioners can steal this allocation strategy: profile your model and shift compute from the “new” network to the “old” frozen one if possible.
3. Noise robustness in MLLMs: This paper provides empirical evidence that MLLM visual features degrade gracefully with noise, suggesting that MLLMs are more robust than commonly assumed. For other tasks like video interpolation or super-resolution, you could similarly inject noisy frames into an MLLM.