Staged Training for Vision-Language Models: Separating Perception from Reasoning

Paper: 2605.20177 Authors: Juncheng Wu, Hardy Chen, Haoqin Tu, Xianfeng Tang, Freda Shi, Hui Liu, Hanqing Lu, Cihang Xie, Yuyin Zhou Categories: cs.CL, cs.CV

The Gap

Current VLMs like GPT-4V and Gemini emphasize long chain-of-thought reasoning for visual tasks. The field assumes that longer reasoning traces lead to better performance. But when these models fail on visual math or perception tasks, researchers have been throwing more reasoning data at the problem—more steps, more explanations, more tokens.

This paper challenges that assumption. The authors find that VLMs aren’t failing because they can’t reason—they’re failing because they can’t see properly in the first place. Existing post-training methods merge perception and reasoning data together, treating them as a single capability. This conflates two fundamentally different skills: extracting information from pixels versus manipulating symbolic representations.

Problem: VLMs underperform on visual tasks
         |
         v
Mainstream assumption: Need better reasoning
         |
         v
Current approach: Merge all training data
         |                (perception + reasoning mixed)
         v
Result: Long reasoning chains, mediocre accuracy
         |
         v
This paper's insight: Perception is the bottleneck
         |
         v
Method: Stage training by capability type
         |        (perception -> visual reasoning -> textual reasoning)
         v
Evidence: +5.2% WeMath, 20.8% shorter reasoning traces
         |
         v
Conclusion: Fix perception first, reasoning becomes easier

The Increment

One sentence: Before this paper, VLM training mixed perception and reasoning data together; after, we know to train perception first with RL, then layer reasoning on top—getting better results with less reasoning.

Core Mechanism

The method decomposes VLM training into three sequential stages, each targeting a specific capability with specialized data. Stage 1 focuses on visual perception—teaching the model to accurately extract information from images (counting objects, reading text, identifying spatial relationships). Stage 2 adds visual reasoning—connecting perceptual observations through logical steps. Stage 3 introduces textual reasoning—pure symbolic manipulation without visual grounding.

Each stage uses different training techniques. Perception benefits most from reinforcement learning (RL) with outcome-based rewards, because caption-style supervision doesn’t provide clear signals for “did you count correctly?” Visual and textual reasoning use supervised fine-tuning (SFT) on chain-of-thought data. The key architectural insight: perception must solidify before reasoning, because reasoning chains built on faulty perception just amplify errors.

Input Image
    |
    v
[Stage 1: Visual Perception]
    |  RL training on:
    |  - Object counting
    |  - OCR tasks
    |  - Spatial relationships
    |  Reward: Correct/Incorrect
    v
Solid perceptual foundation
    |
    v
[Stage 2: Visual Reasoning]
    |  SFT on:
    |  - Multi-hop visual logic
    |  - Diagram interpretation
    v
Visual reasoning capability
    |
    v
[Stage 3: Textual Reasoning]
    |  SFT on:
    |  - Pure math problems
    |  - Symbolic manipulation
    v
Final VLM with decoupled capabilities

Think of it like learning to play chess. You can’t develop opening theory (reasoning) if you can’t reliably distinguish a knight from a bishop (perception). Traditional VLM training is like teaching someone chess notation, tactical patterns, and piece identification all at once—they end up with elaborate plans based on misread positions. This paper says: first drill piece recognition until it’s automatic, then teach tactics, then strategy. Each layer builds on the previous one being solid.

The staged approach also reveals something subtle: when perception improves, models need less reasoning. The paper shows 20.8% shorter reasoning traces with higher accuracy. It’s like how expert chess players “see” moves instantly that novices must calculate—better perception compresses the reasoning required.

Key Concepts

• Capability-based staging: Traditional curriculum learning orders examples by difficulty (easy problems first, hard problems later). This paper introduces a perpendicular dimension: ordering by capability type. Perception must precede reasoning regardless of problem difficulty, because reasoning on bad perception is worse than useless—it’s confidently wrong. The insight is that some capabilities are foundational scaffolds for others, and this dependency structure should guide training order, not just task difficulty.

• RL for perception vs SFT for reasoning: Why does perception benefit from RL while reasoning prefers SFT? Perception tasks have clear ground truth (did you count 7 objects correctly?) but the path to that answer is flexible—the model can attend to different image regions in different orders. RL’s outcome-based rewards let the model discover its own perceptual strategies. Reasoning tasks, by contrast, benefit from explicit step-by-step demonstrations because the logical structure itself is what we want to teach. It’s the difference between “get the right answer however you want” (perception) versus “follow this logical pattern” (reasoning).

• Additive curriculum dimensions: The paper shows that capability-based staging (perception → reasoning) and difficulty-based staging (easy → hard) are orthogonal—you can combine both for additive gains. This suggests training curricula have multiple independent dimensions, like a coordinate system. Most prior work explored one axis (difficulty); this paper adds a second axis (capability type). Future work might discover additional axes (domain, modality, abstraction level) that similarly compose.

Framework Shift

Before (mainstream approach):        After (this paper):

All training data mixed              Stage 1: Perception only
    |                                    |  (RL on visual tasks)
    v                                    v
Single training phase                Stage 2: Visual reasoning
    |                                    |  (SFT on visual CoT)
    v                                    v
VLM with entangled                   Stage 3: Textual reasoning
capabilities                             |  (SFT on math CoT)
    |                                    v
    v                                VLM with decoupled
Long reasoning chains,               capabilities
mediocre accuracy                        |
                                         v
                                     Short reasoning chains,
                                     high accuracy

From treating perception and reasoning as a single monolithic capability to recognizing them as distinct, sequentially-dependent skills—the core shift is architectural: build the foundation before the structure.

Expert Assessment

Problem choice: This is a real gap, not manufactured. The field’s obsession with longer reasoning chains has been a visible trend since GPT-4V, and this paper provides empirical evidence that we’ve been optimizing the wrong bottleneck. The problem sits at a critical juncture—VLMs are being deployed for visual tasks, but their failure modes aren’t well understood. Identifying perception as the limiting factor redirects research effort productively.

Method maturity: The staged training approach is conceptually simple, which is a strength. The authors aren’t proposing exotic architectures or loss functions—they’re reorganizing existing techniques (RL, SFT) based on a clear hypothesis about capability dependencies. However, the paper doesn’t deeply explore why RL specifically helps perception beyond “outcome-based rewards work better.” A more mechanistic understanding of what RL does differently to the model’s visual encoder would strengthen the contribution. The comparison between staged and merged training is thorough, but I’d want to see ablations on stage ordering (what if reasoning comes before perception?) to confirm the dependency claim.

Experimental integrity: Baselines are fair—they compare against merged training with the same total data budget. The results are consistent across multiple VLM architectures (LLaVA, Qwen-VL), which suggests the findings generalize. The 20.8% shorter reasoning traces with higher accuracy is a strong signal that something fundamental changed, not just overfitting to benchmarks. One concern: the paper doesn’t report variance across training runs, so we don’t know if the gains are robust or sensitive to initialization. The benchmark selection (WeMath, RealWorldQA) is appropriate for testing perception, though more diverse visual domains (medical imaging, satellite imagery) would strengthen claims of generality.

Writing quality: The abstract and introduction are clear, but Section 3 (method) buries the key insight—that perception is the bottleneck—under implementation details. The paper would benefit from leading with the diagnostic experiments (showing perception limits performance) before describing the staged training solution. The related work section is thorough but reads like a literature review rather than positioning this work’s unique angle. Figure 2 (the staged training diagram) should appear earlier and be larger—it’s the paper’s conceptual core.

Verdict: weak accept — The core insight (perception bottlenecks VLM performance, stage training by capability type) is valuable and well-demonstrated, but the paper doesn’t fully explain why staged training works mechanistically, limiting its scientific depth.

Takeaways

For practitioners building VLMs: Don’t mix perception and reasoning data in a single training phase. If your model hallucinates visual details, more reasoning data won’t fix it—you need targeted perception training first, and RL works better than caption-based SFT for this.

For curriculum learning researchers: Capability type (perception vs reasoning) is an orthogonal curriculum dimension to difficulty. Check if your domain has similar dependency structures—maybe syntax before semantics in NLP, or motor control before strategy in robotics.

For anyone debugging model failures: When a model fails at a complex task, decompose the failure into capability types (perception, reasoning, memory, etc.) and test each separately. The bottleneck might not be where you think—this paper’s key lesson is that elaborate reasoning on bad perception is worse than simple reasoning on good perception.

Transferable technique: The “stage by capability, not just difficulty” principle applies beyond VLMs. Any system with hierarchical capabilities (where skill B depends on skill A) should consider staged training. The paper’s experimental protocol—comparing staged vs merged training with equal data budgets—is a clean template for testing this in other domains.