Vero: Cracking Open the Black Box of Visual Reasoning with Full-Stack RL

Paper: 2604.04917 Authors: Gabriel Sarch, Linrong Cai, Qunzhong Wang, Haoyang Wu, Danqi Chen, Zhuang Liu Categories: cs.CV, cs.AI, cs.CL

The Gap

Vision-language models like GPT-4V and Gemini show impressive visual reasoning across diverse tasks—charts, diagrams, spatial puzzles, science questions. But their training recipes are locked behind proprietary RL pipelines with undisclosed data. Open-weight models exist, but they either use small-scale RL datasets (tens of thousands of samples) or lack systematic coverage across reasoning types. The result: a 5-10 point performance gap on challenging benchmarks, and no clear path for the research community to close it.

The core question: Can you build a generalist visual reasoner with fully open data and methods?

Problem: Proprietary VLMs dominate visual reasoning
   |
   v
Gap: Open models lack scale + diversity in RL data
   |
   v
Hypothesis: Broad task coverage matters more than task-specific depth
   |
   v
Method: Vero-600K (59 datasets, 6 categories) + task-routed rewards
   |
   v
Evidence: +3.7-5.5 pts over base models, beats existing open RL datasets
   |
   v
Conclusion: Data breadth drives RL scaling for visual reasoning

The Increment

One sentence: Before Vero, open visual reasoning models either used narrow RL data or no RL at all; after Vero, there’s a public 600K-sample recipe that matches proprietary performance across six reasoning categories.

Core Mechanism

Vero’s training pipeline has three stages. First, collect 600K visual reasoning samples from 59 datasets spanning six categories: charts, science diagrams, spatial reasoning, document understanding, general VQA, and math. Second, design task-routed reward models—instead of one monolithic reward, route each sample to a category-specific reward that handles heterogeneous answer formats (multiple choice vs. free-form vs. numeric). Third, run standard RL (they use GRPO, a policy gradient method) to optimize the VLM policy against these rewards.

The key architectural choice: rewards are routed by task category, not learned end-to-end. For charts, the reward checks if the extracted number matches ground truth. For science diagrams, it uses a verifier model trained on that domain. For spatial reasoning, it evaluates geometric correctness. This modular design lets them scale across diverse formats without reward hacking.

Data flow: raw image-question pairs → category classifier → task-specific reward model → RL loss → updated VLM policy. The category classifier is a simple rule-based router (dataset name → category), not a learned model.

Input: (image, question, answer_format)
   |
   v
[Category Router] ---> charts / science / spatial / docs / VQA / math
   |                      |
   v                      v
[Task Reward Model]   [Verifier / Exact Match / Semantic Sim]
   |
   v
[RL Update (GRPO)] ---> Updated VLM Policy

Think of Vero like a restaurant kitchen with specialized stations. You don’t train one chef to handle sushi, pastries, and grilling simultaneously—you route dishes to the right station. The head chef (category router) looks at the order (task type) and sends it to the sushi station (chart reward), pastry station (science reward), or grill (spatial reward). Each station has its own quality control (task-specific verifier). The RL process is the feedback loop: customers rate dishes, and chefs adjust recipes. The insight: trying to train one universal quality checker across all cuisines leads to mediocre results everywhere. Specialization at the reward level, generalization at the policy level.

Key Concepts

• Task-routed rewards: Instead of training one reward model to judge all answer types (multiple choice, free-form text, numeric values), Vero routes each sample to a category-specific reward. Why? Answer formats are heterogeneous. A chart question expects a number; a science diagram expects a label; a spatial puzzle expects a coordinate. A single reward model either overfits to common formats or fails on rare ones. Task routing sidesteps this by using domain-appropriate verifiers. Concrete example: for a chart asking “What’s the value in 2020?”, the reward checks exact numeric match. For a science diagram asking “Label this cell part,” it uses a trained verifier that understands biological terminology. No single model does both well.

• Broad data coverage hypothesis: Vero’s core claim is that RL scaling for visual reasoning requires breadth, not depth. Their ablation: training on one category (e.g., only charts) improves that category by 8 points but barely transfers to others. Training on all six categories improves each by 3-5 points. The math: 6 categories × 3 points > 1 category × 8 points when you care about general reasoning. This contradicts the intuition that specialization wins. The explanation: different reasoning types elicit distinct cognitive patterns (spatial vs. numerical vs. semantic), and the model needs exposure to all patterns to build robust reasoning circuits.

Framework Shift

Before (existing open VLMs):        After (Vero):

[Base VLM]                          [Base VLM]
    |                                   |
    v                                   v
[Small RL data]                     [Vero-600K: 59 datasets]
 (10-50K samples)                    (6 categories, 600K samples)
    |                                   |
    v                                   v
[Single reward model]               [Task-routed rewards]
    |                                   |   |   |   |   |   |
    v                                   v   v   v   v   v   v
[Narrow improvement]               Chart Sci Spatial Doc VQA Math
 (+2 pts on 1-2 tasks)                  |___|___|___|___|___|
                                              |
                                              v
                                        [Broad improvement]
                                         (+3.7-5.5 pts across 30 benchmarks)

From monolithic reward to modular routing, the core shift is recognizing that visual reasoning isn’t one skill but a portfolio of skills requiring specialized feedback.

Expert Assessment

Problem choice: Real gap. The proprietary-open divide in VLMs is a legitimate bottleneck for research. Vero targets the right lever: RL data scale and diversity, not model architecture. This is where the field needs transparency.

Method maturity: Mostly engineering, not algorithmic novelty. Task-routed rewards are sensible but not groundbreaking—it’s careful dataset curation and reward design. The GRPO algorithm is off-the-shelf. The contribution is showing that scale + breadth works, which is valuable but not intellectually surprising. No simpler approach is overlooked; this is the straightforward path.

Experimental integrity: Baselines are fair. They compare against existing open RL datasets (e.g., RLAIF-V) and show Vero-600K wins across categories. The ablation isolating breadth vs. depth is convincing. One concern: they don’t report compute costs or training time, which matters for reproducibility. The 30-benchmark eval suite (VeroEval) is comprehensive, though some benchmarks are saturated (>95% accuracy).

Writing quality: The paper front-loads results and buries dataset construction details in the appendix. Section 3 (data collection) should be expanded—how they filtered 59 datasets, handled annotation quality, and balanced categories is the real recipe. The ablation section (4.3) is the strongest part; it should come earlier. The related work is perfunctory.

Verdict: Weak accept — solid engineering contribution with full open release, but limited conceptual novelty; the value is in execution and transparency, not new ideas.

Takeaways

For practitioners building VLMs: Don’t unify your reward model across task types—route by category and use domain-specific verifiers. The performance gain from specialization outweighs the complexity cost.

For RL researchers: Breadth beats depth in multi-task RL when tasks require qualitatively different reasoning. If your domains don’t transfer well in isolation (Vero shows <1 point cross-category transfer), you need coverage, not concentration.

For dataset builders: The 59-dataset curation process (Appendix A) is a template for large-scale RL data construction. Key insight: balance category representation (100K samples per category) rather than chasing the largest single dataset.

Steal this: The task-routed reward architecture. It’s a simple design pattern that generalizes beyond vision—any multi-domain RL problem with heterogeneous outputs benefits from modular reward routing instead of monolithic reward models.