MMProLong: Training Vision-Language Models for 128K+ Context with Balanced Data

Paper: 2605.13831 Authors: Zhaowei Wang, Lishu Luo, Haodong Duan, Weiwei Liu, Sijin Wu, Ji Luo, Shen Yan, Shuai Peng, Sihang Yuan, Chaoyi Huang Categories: cs.CV

The Gap

Vision-language models (LVLMs) are being pushed into long-context scenarios: analyzing 100-page documents, hour-long videos, multi-turn agent workflows. Existing models like GPT-4V and Gemini handle 32K-128K contexts, but training recipes remain ad-hoc. Prior work focuses on architectural tricks (sparse attention, RoPE scaling) or throws compute at the problem without understanding what data actually teaches long-context ability. The specific gap: no one has systematically ablated data composition for long-context vision-language training. Should you train on 128K-only data? Mix short and long? Use OCR transcripts or VQA? The field is flying blind.

Problem: LVLMs need long context, but training recipes are guesswork
   |
   v
Assumption: Data mixture matters more than architecture tweaks
   |
   v
Method: Systematic ablation of data types, lengths, task ratios
   |
   v
Evidence: Balanced lengths + retrieval-heavy mix beats target-length focus
   |
   v
Conclusion: 5B tokens with right data → 128K training generalizes to 512K

The Increment

One sentence: Before this paper, long-context LVLM training was “throw more 128K data at it”; after, we know balanced-length retrieval data with modest reasoning generalizes beyond training length.

Core Mechanism

The method is continued pre-training on top of Qwen2.5-VL-7B (32K context). The core pipeline: take long documents (PDFs, slides, webpages), convert to images, generate VQA pairs where answers require retrieving information scattered across many pages. Three data types: long-document VQA (questions about content), OCR transcription (just text extraction), and reasoning tasks (math, logic). The training uses standard next-token prediction but with careful data mixing.

The key architectural move is minimal: extend RoPE (rotary position embeddings) from 32K to 128K using YaRN interpolation. No fancy sparse attention, no retrieval-augmented generation modules. The entire contribution is in the data recipe.

Data composition has three knobs: sequence length distribution (how much 32K vs 64K vs 128K), task type ratio (retrieval vs reasoning), and format (VQA vs OCR). The authors train multiple variants, each with 5B tokens, and measure performance on long-document VQA benchmarks plus generalization to 256K and 512K contexts (never seen during training).

Structural metaphor: Think of training a search-and-rescue dog. You could train it only in dense forests (128K-only data), but it won’t generalize to mountains or deserts. Instead, you train across varied terrain (balanced lengths) and focus on the core skill—finding the target (retrieval-heavy tasks)—while adding occasional problem-solving drills (modest reasoning data). The dog learns a generalizable “search” instinct rather than memorizing one environment. Here, the model learns key-information retrieval as a transferable skill, not a length-specific trick.

Key Concepts

• Long-document VQA vs OCR transcription: OCR is “read this page aloud.” VQA is “find the answer to this question somewhere in these 50 pages.” OCR teaches the model to see text but not to search. VQA forces the model to build an internal index of where information lives. The paper shows VQA is 7% better than OCR for long-context ability because retrieval, not transcription, is the bottleneck. Concrete example: OCR learns “page 23 says ‘Q3 revenue: $5M’”; VQA learns “when asked about Q3 revenue, jump to page 23.”

• Balanced-length distribution: Intuition says “train on 128K to get 128K ability.” This paper shows the opposite. Training on 50% 128K + 25% 64K + 25% 32K outperforms 100% 128K data. Why? Because long-context ability is not about handling a specific length—it’s about retrieving key information regardless of where it appears. Balanced data teaches the model “search everywhere” rather than “expect information at position 100K.” It’s like learning to find your keys: practicing in different rooms (varied lengths) beats practicing only in the bedroom (target length).

• Retrieval-heavy mixture: The paper finds 70% retrieval + 30% reasoning beats 50/50 or pure retrieval. Retrieval (VQA) is the primary skill; reasoning (math, logic) adds task diversity to prevent overfitting. Pure retrieval works but slightly underperforms because the model starts memorizing document structures. A small reasoning component acts as regularization, keeping the model general. Think of it as a pianist practicing scales (retrieval) with occasional full pieces (reasoning) to stay musical.

Framework Shift

Before (mainstream approach):        After (this paper):

  [Architecture]                       [Data Recipe]
       |                                    |
   Sparse Attn                         Balanced Lengths
   RoPE Scaling          vs.           Retrieval-Heavy Mix
   Memory Tricks                        VQA > OCR
       |                                    |
   Marginal Gains                      Generalization Leap
   (128K → 128K)                       (128K → 512K)

From architecture-centric scaling to data-centric generalization, the core shift is treating long-context as a retrieval skill learned from diverse examples, not a length-specific capability drilled with target-length data.

Expert Assessment

Problem choice: Real gap. Long-context LVLMs are being deployed (legal doc analysis, video understanding), but training recipes are cargo-culted from language models without vision-specific validation. The field needed this ablation study two years ago.

Method maturity: Refreshingly simple. No architectural novelty—just systematic data ablation. The insight (balanced data beats target-length focus) is non-obvious and challenges intuition. However, the study is limited to one base model (Qwen2.5-VL-7B) and one continued pre-training budget (5B tokens). Generalization to other model families and scales is unproven.

Experimental integrity: Baselines are fair. The ablations are thorough (10+ data mixtures tested). The 256K/512K generalization results are impressive but lack error bars—single-run numbers are risky. The paper claims “maintains short-context capabilities” but only shows aggregate scores, not per-task breakdowns. A deeper analysis of what short-context tasks degrade (if any) would strengthen the claim.

Writing quality: The abstract oversells (“extensive ablations” = 10 runs with 5B tokens each, not extensive by modern standards). Section 3.2 (data construction) is rushed—how exactly are VQA pairs generated? Human-written? Synthetic? The paper says “we generate” but doesn’t specify the pipeline. Rewriting this section with a clear data flowchart would eliminate confusion.

Verdict: Weak accept — solid empirical contribution with actionable insights, but limited scope (one model, one budget) and missing details on data generation reduce impact.

Takeaways

For practitioners:

1. If training long-context vision models, use balanced-length data (mix of 32K/64K/128K) rather than focusing on your target length. The model will generalize better.
2. Prioritize retrieval-heavy tasks (VQA) over transcription tasks (OCR). Retrieval is the bottleneck, not text recognition.
3. Add 20-30% reasoning data (math, logic) to prevent overfitting to document structures, but keep retrieval as the majority.
4. You can likely skip short-data mixing if your long data is instruction-formatted. The paper shows minimal short-context degradation with pure long VQA.

Transferable idea: The “balanced distribution beats target distribution” insight applies beyond vision-language. If training models for any length-dependent task (long code generation, extended dialogue), consider whether you’re teaching a length-specific trick or a generalizable skill. Balanced training data often teaches the latter.