Beyond Language Modeling: How Vision and Language Scale Differently in Multimodal Pretraining

Paper: 2603.03276 Authors: Shengbang Tong, David Fan, John Nguyen, Ellis Brown, Gaoyue Zhou, Shengyi Qian, Boyang Zheng, Théophane Vallaeys, Junlin Han, Rob Fergus Categories: cs.CV

The Gap

Most multimodal models today are language models with vision bolted on. GPT-4V, Gemini, Claude — they all started as language models, then learned to “see” through additional training. This approach works, but it leaves a fundamental question unanswered: what does native multimodal pretraining look like when you don’t inherit language model biases?

The design space is opaque. We don’t know how vision and language data interact during joint training from scratch. We don’t know if they need the same model capacity, the same amount of data, or even the same training objectives. Prior work either fine-tunes pretrained language models on vision tasks or trains small-scale multimodal models without systematic ablations.

This paper strips away the confounding factors. They train everything from scratch, control for compute, and isolate what actually matters.

Problem: Multimodal models inherit language-first biases
   |
   v
Assumption: Vision and language have different scaling properties
   |
   v
Method: From-scratch pretraining + IsoFLOP analysis + MoE architecture
   |
   v
Evidence: Vision needs 10x more data; MoE harmonizes the asymmetry
   |
   v
Conclusion: Native multimodal models require architecture-level solutions

The Increment

One sentence: Before this paper, we built multimodal models by adapting language models; after, we know vision and language scale asymmetrically and need MoE to balance their different appetites.

Core Mechanism

The method has three layers. First, they use Transfusion — a hybrid training framework where language uses next-token prediction (like GPT) and vision uses diffusion (like Stable Diffusion). This lets them train on text, images, videos, and image-text pairs in one unified model without forcing everything through the same objective.

Second, they introduce Representation Autoencoder (RAE) as the visual backbone. RAE compresses images into discrete tokens that work for both understanding (classification, captioning) and generation (image synthesis). Think of it as a universal translator between raw pixels and the model’s internal language.

Third, they run IsoFLOP experiments — training different model configurations with the same total compute budget. This reveals scaling laws: how performance changes as you vary data mix, model size, and architecture. The key finding: vision needs vastly more data than language to reach comparable performance.

Input Data Stream
   |
   +---> [Text] ---> Next-Token Prediction ---> Language Loss
   |
   +---> [Images/Video] ---> RAE Encoder ---> Discrete Tokens
                                |
                                v
                          Diffusion Process ---> Vision Loss
                                |
                                v
                          Combined Gradient ---> Model Update
                                |
                                v
                          [Shared Transformer + MoE Layers]

Think of this like running a restaurant with two kitchens. The language kitchen (next-token prediction) is efficient — it can serve many customers with modest ingredients. The vision kitchen (diffusion) is demanding — it needs premium ingredients in bulk to produce quality dishes. RAE is the prep station that standardizes ingredients from both suppliers. The MoE architecture is your staffing solution: you hire specialist chefs for each kitchen, but they share the dining room and back-office infrastructure. Without MoE, you’d either overstaff the language kitchen (wasting capacity) or understaff the vision kitchen (poor quality). The asymmetry isn’t a bug — it’s the nature of the cuisines.

Key Concepts

• IsoFLOP Analysis: Imagine you have a fixed energy budget to train models. IsoFLOP means you compare different designs (bigger model with less data vs. smaller model with more data) while keeping total compute constant. It’s like comparing cars by miles-per-gallon at the same trip distance, not just top speed. This reveals true efficiency: which design extracts the most performance from the same resources? In this paper, IsoFLOP analysis exposed that vision models need to “see” 10x more examples than language models need to “read” to achieve similar capability gains.

• Representation Autoencoder (RAE): Most vision models specialize — CLIP for understanding, diffusion models for generation. RAE does both with one representation. It compresses an image into discrete tokens (like turning a photo into a short code), then can either decode those tokens back into pixels (generation) or feed them to a classifier (understanding). The trick: the compression is learned to preserve information useful for both tasks. Concrete example: RAE encodes a cat photo into tokens [42, 891, 3, …], which can reconstruct the cat OR be classified as “cat” — same tokens, dual purpose.

• Scaling Asymmetry: Language and vision don’t scale the same way. If you double your language data, you get predictable improvements. But vision is greedier — it needs 10x more data to get the same relative improvement. Why? Language has compositional structure (words combine systematically), while vision has exponential variation (lighting, angles, occlusion, textures). It’s like learning chess vs. learning to recognize faces: chess has finite rules you can master with moderate practice, but faces have infinite variation requiring massive exposure.

Framework Shift

Before (mainstream approach):        After (this paper):

[Pretrained LLM]                     [From-Scratch Training]
       |                                      |
       v                                      v
  Add Vision Adapter              [Transfusion Framework]
       |                            /                  \
       v                           v                    v
  Fine-tune on                [Language]            [Vision]
  Image-Text Pairs            Next-Token            Diffusion
       |                      Prediction               |
       v                           \                  /
  Multimodal Model                  v                v
  (language-centric)           [Unified Model + MoE]
                               (balanced modalities)

From bolting vision onto language models to designing architectures that respect each modality’s native scaling properties, the core shift is treating multimodality as a first-class design problem rather than an adaptation problem.

Expert Assessment

Problem choice: This is a real gap. The field has been building multimodal models pragmatically — start with what works (language models) and extend. But we’re hitting limits. Models like GPT-4V are impressive yet clearly language-biased in their reasoning. Understanding native multimodal scaling is foundational work that will age well. It sits at an inflection point where compute budgets allow from-scratch experiments that were previously prohibitive.

Method maturity: Mostly solid engineering with one clever insight (RAE for unified representation). The Transfusion framework isn’t novel — it’s borrowed from prior work. The MoE application is straightforward. The real contribution is the systematic ablation study and IsoFLOP analysis. They’re not inventing new techniques; they’re rigorously measuring what matters. That said, the 10x data asymmetry finding feels almost too clean — I’d want to see if it holds across different vision domains (medical imaging, satellite imagery) or if it’s specific to natural images.

Experimental integrity: The baselines are fair and the ablations are thorough. They control for compute, which is rare and commendable. However, the paper doesn’t deeply explore failure modes. What happens when you push MoE scaling further? Do the modality-specific experts actually specialize, or do they collapse into redundancy? The scaling laws are derived from limited compute ranges — extrapolation to GPT-4 scale is speculative. Also, they claim “world modeling” emerges naturally, but the evidence is thin — a few cherry-picked examples, not systematic evaluation.

Writing quality: The paper front-loads claims without sufficient setup. The abstract promises “four key insights” but doesn’t explain why these specific four matter. The related work section is perfunctory — it lists prior work without critically positioning this contribution. The experimental section is dense with numbers but light on intuition. If they rewrote the introduction to tell a story (why does native multimodal pretraining matter? what’s at stake?), the paper would be far more compelling. The conclusion oversells — “paving the way for truly unified multimodal models” is premature given the limited scale.

Verdict: weak accept — Solid empirical work that advances our understanding of multimodal scaling, but lacks the conceptual depth or scale to be a landmark contribution.

Takeaways

The 10x data asymmetry is the actionable insight. If you’re building a multimodal system, don’t assume vision and language need equal resources. Budget accordingly: vision needs more data, language needs more model capacity. The MoE solution is practical — you can scale modalities independently without wasting parameters.

RAE’s dual-purpose representation is worth stealing. If you’re designing any encoder, ask: can this representation serve multiple downstream tasks? Unified representations reduce engineering complexity and often improve both tasks through shared structure.

The IsoFLOP methodology transfers. Anytime you’re comparing architectures, control for total compute. It’s the only fair comparison and reveals true efficiency gains versus just throwing more resources at the problem.

One warning: the “world modeling” claim is oversold. Don’t expect emergent capabilities from general multimodal training without task-specific design. The paper shows correlation, not causation.