Unbundling Multimodal Vision Encoding: Why Pretrained Priors Are Optional but Dedicated In-Fusion Capacity Is Key

Paper: 2611.17334
Authors: Ellis Brown, Chris Hoang, Shengyi Qian, Shang-Wen Li, Mike Lewis, Marjan Ghazvininejad, Luke Zettlemoyer, Lucas Beyer, Mengye Ren, Jakob Verbeek, Saining Xie, Hu Xu (Meta FAIR, New York University)
Categories: cs.CV, cs.LG

The Gap

How Multimodal Large Language Models (MLLMs) should ingest visual inputs is the most contested architectural debate in foundation models:

• Mainstream Recipe (e.g., LLaVA, Qwen-VL, Pixtral): Prepends an independently pretrained Vision Transformer (e.g., CLIP, SigLIP, DINOv2) to extract patch tokens before projecting them into a frozen or fine-tuned language backbone.
• Joint Training Recipe (e.g., Chameleon, PaLI): Keeps a separate vision encoder but trains it from scratch concurrently with the language backbone.
• Encoder-Free Recipe (e.g., Fuyu, Gemma 3): Discards the vision encoder entirely, feeding raw linear patch embeddings directly into the shared autoregressive transformer.

Comparing these paradigms across published models has historically been impossible because existing models differ across data distributions, training recipes, tokenizers, and parameter budgets.

Crucially, the authors observe that standard vision encoders bundle three orthogonal design choices:

1. The Visual Prior: Learned representations imported from upstream pretraining (e.g., contrastive language-image learning).
2. Dedicated Vision Capacity: A block of parameters allocated exclusively to processing vision tokens without parameter competition from text.
3. Pre-Fusion Placement: Placing this visual capacity strictly before the multimodal fusion boundary (the first layer where vision and text interact).

                  THE THREE BUNDLED CHOICES OF VISION ENCODERS

  [Standard Pretrained Vision Encoder (e.g. CLIP / SigLIP)]:
  Image Patches ──> [Pretrained ViT (0.4B)] ──> Projector ──> [Multimodal Backbone]
                    ├─ 1. Upstream Visual Prior?  (CLIP pretraining)
                    ├─ 2. Dedicated Vision Capacity? (0.4B vision-only params)
                    └─ 3. Pre-Fusion Placement?  (Runs entirely BEFORE text fusion)

  The Critical Question: Which of these three choices actually drives performance?
                         Can we eliminate the separate encoder without sacrificing quality?

---

The Increment

One sentence: Through a strictly controlled, lifetime-compute-matched framework spanning backbones from 0.6B to 6.4B parameters, the authors prove that upstream visual priors provide zero benefit over joint training from scratch, but dedicated vision capacity is mandatory; crucially, this capacity can be relocated in-fusion into the first 8 layers of the language backbone, matching pre-fusion encoders while collapsing the dual-model serving stack and slashing Time-To-First-Token (TTFT).

                      UNBUNDLING THE VISION ENCODER PARADIGM

  Question 1: Is Upstream Pretraining Needed?
  [Pretrained Encoder] vs [Scratch Encoder] (Matched Lifetime FLOPs)
  ──> Result: From-scratch encoder MATCHES or BEATS pretrained encoder!
              Pretraining FLOPs are better spent directly on multimodal training.

  Question 2: Is Dedicated Vision Capacity Needed?
  [Scratch Encoder (1.7B + 0.4B)] vs [Encoder-Free (2.1B Unified Backbone)]
  ──> Result: Encoder-free trails badly on VQA (-5.4% at 1.7B, requires 3.0x-4.4x FLOPs).
              Vision tokens demand dedicated parameters!

  Question 3: Must Capacity Precede Multimodal Fusion?
  [Pre-Fusion Encoder (0.4B)] vs [In-Fusion Vision Modules in Layers 1-8]
  ──> Result: In-fusion matches separate encoder across all VQA benchmarks,
              unifies the serving runtime, and reduces Time-to-First-Token (TTFT)!

---

Core Framework & Experimental Protocol

1. The Lifetime Compute-Matched Protocol

To eliminate confounding variables, every model is trained from scratch on common data:

• Text Data: Nemotron-CC v2/v2.1.
• Image-Text Data: MetaCLIP 1.2 English pairs (re-captioned using Qwen2.5-VL-32B).
• Multimodal Data Ratio: 70% text, 30% image-text.
• Lifetime Compute: Every comparison strictly matches the cumulative FLOPs of all training stages (including upstream encoder pretraining).

                    COMPARISON OF THE FOUR DISENTANGLED RECIPES

  Recipe                 Visual Prior?   Dedicated Vision Capacity?   Pre-Fusion Placement?
  ─────────────────────────────────────────────────────────────────────────────────────────
  1. Pretrained Encoder      YES                    YES                       YES
  2. Scratch Encoder         NO                     YES                       YES
  3. Encoder-Free            NO                     NO                        NO
  4. In-Fusion (Proposed)    NO                     YES                       NO (Layers 1-8)

2. Dissecting the Three Choices

Choice 1: Upstream Visual Prior vs. From-Scratch Training

The authors compared four distinct pretraining objectives (MetaCLIP, WebSSL, Cap, CapPa) against a from-scratch encoder under a fixed $1.04 \times 10^{22}$ FLOP lifetime budget:

• Spending up to 15% of lifetime compute on encoder pretraining performed no better than allocating 100% of compute to end-to-end multimodal training.
• When pretraining consumed 60% of lifetime compute, multimodal VQA accuracy collapsed.
• Even when encoder pretraining compute was treated as free (zero-cost), the from-scratch encoder remained fully competitive with MetaCLIP.

Choice 2: The Fallacy of Encoder-Free Architectures

Proponents of encoder-free models claim that projecting patches directly into a larger unified backbone is optimal. The authors enlarged the encoder-free backbone by the exact parameter count of the encoder (1.7B $\to$ 2.1B):

• Visual Understanding Trails: Encoder-free models scored 39.0% on Average VQA versus 44.4% for the scratch encoder.
• Compute Inefficiency: The encoder-free architecture requires $3.0\times$ to $4.4\times$ more training FLOPs to achieve parity with an encoder-augmented model on visual question answering.
• Reason: Without dedicated vision parameters, vision tokens and text tokens compete destructively for shared feed-forward network (FFN) capacity, diluting fine-grained perceptual representations.

Choice 3: Pre-Fusion vs. In-Fusion Placement

If dedicated vision parameters are required, do they need to reside in a separate pre-fusion Vision Transformer? The authors reallocated the 0.4B vision parameters directly inside the backbone as vision-only FFN and attention branches:

• Where to Place: Placing vision-only modules in the first 8 layers (Layers 1–8) achieved 45.0% VQA, matching or slightly exceeding the pre-fusion scratch encoder (44.4%).
• Spreading the capacity evenly across all layers (37.6%) or placing it in late layers (35.1%) failed completely.
• Mechanism: Vision tokens require specialized processing early, while they are still close to raw patch signals; once fused into abstract semantic representations, shared transformer layers handle multimodal reasoning efficiently.

---

Comprehensive Benchmark Results

1. Architectural Ablation across VQA and Text Benchmarks

All models evaluate 0.4B dedicated vision capacity paired with a 1.7B backbone at matched $1.04 \times 10^{22}$ FLOP budget:

Architecture Variant	Visual Capacity	Placement	Avg. VQA (%) $\uparrow$	General VQA	Knowledge VQA	OCR & Charts	Visual Reason	Avg. Text (%) $\uparrow$
Scratch Encoder	Dedicated (0.4B)	Pre-Fusion	44.4	57.2	35.7	38.2	46.5	49.7
Encoder-Free	None (Shared)	—	39.0	50.7	35.0	25.2	45.0	50.7
In-Fusion (Every Layer)	Dedicated (0.4B)	Narrow All	39.4	46.4	33.4	31.3	46.6	50.7
In-Fusion (Late Layers)	Dedicated (0.4B)	Layers 21–28	35.1	42.6	31.4	23.3	43.3	49.8
In-Fusion (Early Layers)	Dedicated (0.4B)	Layers 1–8	45.0	57.0	36.7	36.0	50.4	50.4
Shared FFN Widening	All Tokens	Layers 1–8	40.1	51.5	34.1	29.6	45.1	51.2

Key Findings:

• In-Fusion (Early) matches the Pre-Fusion Scratch Encoder across all dimensions and delivers a massive +10.8 percentage point gain in OCR/chart understanding over Encoder-Free (36.0% vs. 25.2%).
• Reserving parameters strictly for vision tokens is critical: widening shared FFNs for all tokens yields only 40.1% VQA.

2. Scaling Ladder from 0.6B to 6.4B Parameters

Across the entire scaling trajectory (growing vision capacity proportionally from 0.1B to 1.6B parameters), In-Fusion consistently tracks or outperforms the Pre-Fusion Encoder, while Encoder-Free remains permanently handicapped:

                      VQA SCALING CURVES (0.6B TO 6.4B)

  VQA Score (%)
    60 ┌                                               ▲ In-Fusion (58.2%)
       │                                            ▲ Scratch Encoder (57.9%)
    50 │                             ▲ In-Fusion
       │                      ▲ Scratch Encoder
    40 │               ▲ In-Fusion
       │        ▲ Scratch Encoder        ■ Encoder-Free (Trails by ~5-6 points!)
    30 └─────────────────────────────────────────────────────────────────────
       0.6B           1.7B              4.0B                     6.4B
                                Model Scale (Backbone + Vision)

---

Practical Implications

1. Abandon Separate Vision Encoder Pretraining: Foundation model labs do not need to maintain complex, two-stage pretraining pipelines (e.g., training SigLIP on billions of image-text pairs before training the LLM). End-to-end multimodal joint training from scratch is compute-optimal.
2. The End of the Two-Model Serving Bottleneck: Separate vision encoders force serving systems (vLLM, SGLang) to manage two distinct neural network forward passes, pipeline stalls, and cross-model tensor copies. In-Fusion consolidates vision processing into the main transformer graph, dramatically reducing Time-to-First-Token (TTFT).
3. The Definitive Verdict on Encoder-Free Models: Pure encoder-free vision modeling (feeding linear patch embeddings into a homogeneous transformer) is fundamentally suboptimal. Perceptual token processing requires dedicated parameter capacity in the initial layers.

                     视觉编码器深度捆绑的三大核心设计维度

  【传统外挂视觉编码器架构 (如 CLIP / SigLIP)】：
  输入图像 ──> [独立预训练 ViT 编码器 (0.4B)] ──> 投影层 ──> [多模态语言模型主干]
               ├─ 1. 是否必须具备上游预训练视觉先验？ (对比学习/分类先验)
               ├─ 2. 是否必须具备专门处理视觉的参数容量？ (0.4B 独立视觉参数)
               └─ 3. 这部分视觉算力是否必须严格置于大模型融合之前？ (完全独立前置)

  核心追问：究竟是哪个因素在真正起作用？我们能否彻底干掉外挂 ViT，实现极简部署？

                        解构多模态视觉编码的三大实验定论

  问题 1：视觉编码器真的需要独立预训练吗？
  【预训练编码器】 vs 【从零联合训练编码器】（严格对齐全生命周期总 FLOPs）
  ──> 结论：从零联合训练全面对齐甚至超越预训练！上游预训练算力不如直接砸给多模态阶段。

  问题 2：视觉编码需要专用参数容量吗？
  【从零编码器 (1.7B + 0.4B)】 vs 【无编码器架构 (统一扩容至 2.1B 主干)】
  ──> 结论：无编码器方案在视觉问答上全面溃败（OCR暴跌13个点，需3.0~4.4倍算力追平）！
            视觉特征必须拥有专属参数，不能与文本共享 FFN 争抢容量。

  问题 3：专用视觉算力必须置于大模型之前吗？
  【前置独立 ViT (0.4B)】 vs 【内置于大模型前 8 层的视觉专属模块 (In-Fusion)】
  ──> 结论：In-Fusion 在所有 VQA 任务上全面超越前置独立编码器！
            不仅彻底消灭了双模型架构，更大幅降低首字响应延迟（TTFT）。

                         四种解构实验路线机制对比表

  架构方案路线           视觉先验 (Prior)   专用视觉容量 (Capacity)  融合前置位置 (Pre-Fusion)
  ─────────────────────────────────────────────────────────────────────────────────────────────
  1. 预训练编码器路线         有                     有                      是 (前置独立 ViT)
  2. 从零训练编码器路线       无                     有                      是 (前置独立 ViT)
  3. 无编码器原生路线         无                     无 (全与文本共享)       否 (线性映射入主干)
  4. In-Fusion 内置路线      无                     有 (专属视觉分支)       否 (嵌入主干前 8 层)

                    0.6B 到 6.4B 视觉问答扩展轨迹图示

  VQA 综合准确率 (%)
    60 ┌                                               ▲ In-Fusion (58.2%)
       │                                            ▲ 从零独立编码器 (57.9%)
    50 │                             ▲ In-Fusion
       │                      ▲ 从零独立编码器
    40 │               ▲ In-Fusion
       │        ▲ 从零独立编码器         ■ 无编码器方案 (全程落后 5~6 个百分点!)
    30 └─────────────────────────────────────────────────────────────────────
       0.6B           1.7B              4.0B                     6.4B
                                模型总体参数规模 (主干 + 视觉)