OpenVLThinkerV2: Gaussian GRPO for Balanced Multimodal Reasoning

Paper: 2604.08539 Authors: Wenbo Hu, Xin Chen, Yan Gao-Tian, Yihe Deng, Nanyun Peng, Kai-Wei Chang Institution: University of California, Los Angeles (UCLA) Categories: cs.CV

Abstract

GRPO has become the de facto RL objective for multimodal models, but extending it to open-source generalist models faces two challenges: extreme variance in reward topologies across diverse visual tasks, and balancing fine-grained perception with multi-step reasoning. OpenVLThinkerV2 introduces Gaussian GRPO (G²RPO), which replaces linear scaling with non-linear distributional matching via optimal transport, forcing advantage distributions to converge to N(0,1) for inter-task gradient equity.

Key Contributions

• G²RPO: Non-linear distributional matching via 1D optimal transport that maps any task’s advantage distribution to N(0,1)
• Task-level shaping: Response length and entropy shaping mechanisms to balance perception and reasoning
• SOTA performance: 71.6% on MMMU, 79.5% on MathVista, surpassing GPT-4o and Gemini 2.5 Pro on multiple benchmarks

The Gap: Linear Scaling Fails Multi-Task Vision

Standard GRPO normalizes advantages using sample-wise standard deviation, which creates intra-task imbalance by favoring low-variance rollouts. Dr.GRPO removes this normalization but causes inter-task imbalance where high-variance tasks dominate gradients. EMA-GRPO uses task-wise moving averages, but linear transformations only match first two moments (mean and variance) while preserving distributional shape—leaving optimization vulnerable to:

• Heavy-tail outliers: A single anomalous reward inflates EMA variance, suppressing normal responses
• Bimodal distributions: Binary rewards (math tasks) create step-functions with jagged gradients
• Scale mismatch: Dense grounding rewards (IoU scores) get minimal updates compared to sparse math rewards

The Increment: Optimal Transport for Advantage Normalization

G²RPO abandons scalar standardization for non-linear distributional matching. Instead of normalizing via mean/variance, it uses 1D Optimal Transport to strictly map empirical reward distributions to the standard normal N(0,1):

The Algorithm

1. Rank rewards: Compute uniform probability p_i = (rank(R_i) - 0.5) / N
2. Quantile mapping: Map to inverse CDF of N(0,1): Ψ(R_i) = Φ⁻¹(p_i) = √2 · erfinv(2p_i - 1)
3. Tie-breaking: Assign mean of target quantiles to identical rewards for consistent learning signals

This closed-form solution via CDFs is computationally efficient and mathematically guarantees:

• Outlier robustness: Caps extreme rewards at highest quantile
• Symmetric updates: Converts bimodal step-functions to smooth Gaussian tails
• Inter-task equity: Every task’s advantage distribution converges to N(0,1)

Core Mechanism: Balancing Perception and Reasoning

Beyond G²RPO, OpenVLThinkerV2 introduces task-level shaping to balance vision-centric tasks (OCR, grounding) with reasoning-centric tasks (math VQA):

Response Length Shaping

Observation: Reasoning tasks converge to longer chains, while vision tasks optimize toward concise outputs. Solution: Trapezoidal reward envelope with task-specific thresholds (L_min, L_low, L_high, L_max) that:

• Encourages extended reasoning for complex queries
• Enforces direct outputs for visual grounding
• Mitigates hallucinations by preventing unnecessary elaboration

Entropy Shaping

Observation: Reasoning tasks suffer entropy explosion (incoherent tokens), vision tasks suffer entropy collapse (premature exploitation). Solution: Margin-based penalty that bounds entropy within optimal zone:

L_ent_reg = max(0, H_task - H_max) + max(0, H_min - H_task)

This prevents both over-exploration and over-exploitation across diverse task topologies.

Framework Shift: From Linear Moments to Distributional Matching

Traditional normalization (GRPO, EMA-GRPO) relies on linear transformations that preserve distributional shape. G²RPO fundamentally shifts to distributional matching via optimal transport:

Approach	Method	Inter-Task Equity	Outlier Robustness	Symmetry
GRPO	Sample-wise std	❌ Intra-task bias	❌ Vulnerable	❌ Skewed
EMA-GRPO	Task-wise EMA	⚠️ Momentum lag	❌ Vulnerable	❌ Skewed
G²RPO	Optimal Transport	✅ Guaranteed	✅ Capped	✅ Gaussian

The key insight: matching higher-order distributional structure (not just mean/variance) eliminates structural pathologies inherent to multi-domain visual RL.

Expert Assessment: Comprehensive Benchmark Results

Evaluated across 18 benchmarks in 6 task categories:

Visual Reasoning:

• MMMU: 71.6% (vs GPT-4o 70.7%)
• MMBench: 88.2%
• MathVista: 79.5%
• MathVerse: 65.8%

Document Understanding:

• OCRBench: 911 (vs GPT-5 810, Gemini 2.5 Pro 866)
• DocVQA: 96.7%
• InfoVQA: 86.4%

Spatial Reasoning:

• EmbSpatial: 83.1% (vs GPT-5 82.9%, Gemini 2.5 Pro 79.1%)
• RefSpatial: 44.6%
• RoboSpatial: 63.2%

Grounding:

• RefCOCO: 93.4%
• RefCOCO+: 88.2%
• RefCOCOg: 90.4%

Ablation study confirms G²RPO provides the largest gain (+5.6% average across 6 domains), with entropy shaping (+0.5%) and length shaping (+0.9%) providing complementary improvements.

Takeaways

1. Optimal transport > linear scaling: Non-linear distributional matching eliminates structural pathologies in multi-task visual RL
2. Task-level shaping works: Simple trapezoidal envelopes for length/entropy accelerate convergence without delicate tuning
3. Generalist models are viable: Single model achieves SOTA across perception, reasoning, and hybrid tasks
4. Beyond vision: G²RPO’s ability to harmonize divergent reward topologies extends to any multi-domain RL (coding, GUI tasks)
5. Efficient closed-form: 1D optimal transport via CDFs adds minimal computational overhead

The shift from moment-matching to distributional matching represents a fundamental advance in multi-task RL stability, with implications far beyond multimodal models.

L_ent_reg = max(0, H_task - H_max) + max(0, H_min - H_task)