Faithful GRPO: Improving Visual Spatial Reasoning via Constrained Policy Optimization

Paper: 2604.08476 Authors: Sai Srinivas Kancheti, Aditya Kanade, Rohit Sinha, Vineeth N Balasubramanian, Tanuja Ganu (IIT Hyderabad, Microsoft Research) Categories: cs.CV

The Accuracy-Faithfulness Gap

Multimodal reasoning models (MRMs) trained with Reinforcement Learning with Verifiable Rewards (RLVR) show improved accuracy on visual reasoning benchmarks. DeepSeek-R1 demonstrates that a two-stage recipe—supervised finetuning (SFT) followed by Group Relative Policy Optimization (GRPO)—yields strong reasoning performance.

But accuracy alone is incomplete. A model that generates the correct answer despite flawed, contradictory, or visually ungrounded reasoning may have learned to exploit shortcuts and biases. This makes it untrustworthy for real-world use.

The problem: Accuracy gains often come at the cost of reasoning quality. Generated Chain-of-Thought (CoT) traces are frequently inconsistent with the final answer and poorly grounded in visual evidence.

Two Failure Modes

Faithful GRPO identifies two distinct failure modes in CoT traces generated by RLVR-trained MRMs:

1. Logical Inconsistency

The reasoning trace argues toward one conclusion, but the model then abruptly flips its prediction to a different final answer.

Example (spatial reasoning):

• Question: “Which object is closer to the camera, the lamp (red box) or the box (blue box)?”
• GRPO reasoning: “The red box is closer to the camera because it is positioned lower in the image and appears larger… The box in the blue box is smaller… it is farther from the camera. Therefore, the lamp is the correct answer.”
• GRPO answer: box (contradicts reasoning)
• Faithfulness: 33%, Inconsistent

The model reasons toward “lamp” but answers “box”. The reasoning and answer are logically inconsistent.

2. Visual Ungroundedness

Individual reasoning steps describe objects, attributes, or spatial relationships that are inaccurate with respect to the visual content.

Example (counting paths):

• Question: “How many paths are in the image?”
• GRPO reasoning: “The image shows a large building with two tall minarets and a dome, surrounded by trees. The area appears to be a courtyard or plaza, but there are no visible paths leading through it. The entire image has been examined… no paths have been found…”
• GRPO answer: 1.0 (contradicts reasoning)
• Faithfulness: 50%, Inconsistent

The model claims there are no visible paths, then answers “1.0”. The reasoning is visually ungrounded—there is a visible path in the image.

Key insight: A trace can be logically consistent yet visually ungrounded, or well-grounded yet inconsistent with its own answer. Both dimensions matter for trustworthy reasoning.

Measuring Reasoning Quality

Faithful GRPO characterizes CoT reasoning quality along two complementary axes:

Logical Consistency

Does the CoT entail the final answer? Evaluated by an LLM judge that determines whether the final answer follows logically from the reasoning trace.

Consistency reward: RC(o) ∈ $\{0, 1\}$

• 1 if reasoning’s conclusion matches final answer
• 0 if inconsistent
• Masked to samples where answer is correct (consistency only meaningful for non-trivial answers)

Visual Grounding

Does each reasoning step accurately describe objects, attributes, and spatial relationships in the image?

Semantic grounding reward: RS(o) ∈ [0, 1]

• Decomposes reasoning trace into sentences
• Filters out trivial non-visual sentences (meta-reasoning, planning, hedging)
• Scores each remaining sentence via VLM judge that receives image, question, and reasoning context
• Averages scores across sentences

Spatial grounding reward: RB(o) ∈ [0, 1]

• For tasks with bounding box annotations
• Measures IoU between predicted bounding boxes and ground-truth regions
• Captures spatial localization accuracy

Faithful GRPO: Constrained Policy Optimization

Standard GRPO maximizes task accuracy (answer correctness). Faithful GRPO (FGRPO) maximizes task accuracy subject to minimum thresholds on consistency and grounding rewards.

Formulation

FGRPO treats consistency and grounding as hard constraints enforced via Lagrangian dual ascent:

Objective:

• Maximize: E[Rtask(o)]
• Subject to: E[RC(o)] ≥ τC (consistency threshold)
• Subject to: E[RG(o)] ≥ τG (grounding threshold)

where RG combines semantic and spatial grounding rewards.

Lagrangian Dual Ascent

Lagrange multipliers (λC, λG) adaptively increase pressure on violated constraints and decrease it on satisfied ones, removing the need for manual reward weight tuning.

Advantage computation:

• Decouple normalization: compute per-reward advantages separately
• Combine with Lagrangian weights: Â = Âtask + λC · ÂC + λG · ÂG
• Update multipliers: increase λ when constraint violated, decrease when satisfied

Key difference from prior work:

• MO-GRPO and GDPO use fixed or conditional weights
• FGRPO uses adaptive Lagrangian multipliers that enforce constraint thresholds
• Treats consistency and grounding as prerequisites for trustworthy reasoning, not just additional reward terms

Experimental Setup

Backbones: Qwen2.5-VL-7B and 3B

Training:

• Stage 1 (SFT): 45K CoT traces from SAT, VGR, VisCoT datasets, generated via MCTS with Qwen2.5-VL-72B teacher
• Stage 2 (RL): 49K samples with difficulty-based filtering, including 13K from TreeVGR-RL37K for bounding-box supervision

Evaluation: Seven spatial reasoning datasets

• SAT (Spatial Aptitude Test)
• VGR (Visual Grounding Reasoning)
• VisCoT (Visual Chain-of-Thought)
• Plus four additional benchmarks

Baselines:

• GRPO-Task: Standard GRPO with task reward only
• ViGoRL-Spatial: MCTS-generated point-grounded CoT traces
• TreeVGR: Dual IoU-based rewards for localization and reasoning

Results

Reasoning Quality (Qwen2.5-VL-7B)

Model	Inconsistency Rate	Semantic Grounding	Spatial Grounding
Qwen2.5-VL-7B (base)	18.2%	0.72	0.58
GRPO-Task	24.5%	0.69	0.61
FGRPO	1.7%	0.82	0.74

Dramatic improvement:

• Inconsistency rate: 24.5% → 1.7% (14.4× reduction)
• Semantic grounding: +13% absolute improvement
• Spatial grounding: +13% absolute improvement

Standard GRPO (GRPO-Task) actually increases inconsistency rate compared to the base model (24.5% vs 18.2%), confirming that optimizing for task accuracy alone degrades reasoning quality.

Answer Accuracy

Model	Average Accuracy
Qwen2.5-VL-7B (base)	64.17%
GRPO-Task	65.17%
FGRPO	66.84%

FGRPO improves both reasoning quality and answer accuracy, demonstrating that faithful reasoning and accurate answers are complementary objectives, not competing ones.

Comparison to Contemporary MRMs

Model	Inconsistency Rate	Accuracy
ViGoRL-Spatial	19.3%	64.8%
TreeVGR	21.7%	65.4%
FGRPO	1.7%	66.8%

FGRPO substantially outperforms contemporary MRMs on both reasoning quality and accuracy.

Ablation: Constraint vs. Reward

Configuration	Inconsistency	Accuracy
GRPO-Task (baseline)	24.5%	65.17%
GRPO + weighted rewards	18.9%	65.43%
FGRPO (constraints)	1.7%	66.84%

Adding consistency and grounding as weighted reward terms helps (18.9% inconsistency) but doesn’t match the effectiveness of treating them as hard constraints (1.7% inconsistency).

Key Insights

1. Accuracy alone is insufficient: Standard GRPO improves accuracy but degrades reasoning quality. Inconsistency rate increases from 18.2% (base model) to 24.5% (GRPO-Task).

2. Constraints > Weighted Rewards: Treating consistency and grounding as hard constraints via Lagrangian dual ascent is more effective than adding them as weighted reward terms.

3. Faithfulness enables accuracy: FGRPO improves both reasoning quality (1.7% inconsistency) and answer accuracy (66.84%), demonstrating that faithful reasoning and correct answers are complementary.

4. Two dimensions of faithfulness: Logical consistency and visual grounding are complementary. A trace can be consistent yet ungrounded, or grounded yet inconsistent. Both matter.

5. Adaptive constraint enforcement: Lagrangian multipliers automatically adjust pressure on violated constraints, removing the need for manual reward weight tuning.

Implications

For multimodal reasoning researchers: Accuracy is not enough. Reasoning quality—logical consistency and visual grounding—must be measured and optimized. FGRPO demonstrates that constrained policy optimization can enforce both.

For RLVR practitioners: When training with verifiable rewards, consider what you’re optimizing for. Task accuracy alone can lead to shortcut learning and degraded reasoning quality. Constraints on reasoning quality can improve both faithfulness and accuracy.

For trustworthy AI: Models that reason correctly are more trustworthy than models that answer correctly via flawed reasoning. FGRPO shows that we can have both—faithful reasoning enables better answers.

Limitations and Future Work

• Computational cost: FGRPO requires additional reward models (LLM judge for consistency, VLM judge for semantic grounding). This increases training cost compared to standard GRPO.

• Constraint thresholds: FGRPO requires setting consistency and grounding thresholds (τC, τG). The paper uses fixed thresholds; adaptive threshold selection remains future work.

• Generalization beyond spatial reasoning: Evaluation focuses on spatial reasoning tasks. Generalization to other multimodal reasoning domains (temporal, causal, mathematical) requires further study.

• Judge reliability: Consistency and grounding rewards depend on LLM/VLM judges. Judge errors propagate to training. More robust judge designs could improve FGRPO.

Faithful GRPO demonstrates that constrained policy optimization can enforce reasoning quality in multimodal models, reducing inconsistency rates by 14× while improving answer accuracy. The key insight: treat consistency and grounding as prerequisites for trustworthy reasoning, not optional reward terms.