GRPO-VPS: Verifiable Process Supervision for Effective Reasoning

Paper: 2604.20659 Authors: Jingyi Wang, Lei Zhu, Tengjin Weng, et al. (ICLR 2026) Categories: cs.AI, cs.LG

Abstract

Building on Group Relative Policy Optimization (GRPO), this work introduces model-free and verifiable process supervision via probing the model’s belief in the correct answer throughout its reasoning trajectory. By segmenting generation into discrete steps and tracking conditional probability of the correct answer appended at each segment boundary, they efficiently compute interpretable segment-wise progress measurements.

Problem with GRPO

GRPO eliminates critic models but suffers from indiscriminate credit assignment for intermediate steps, which:

• Limits ability to identify effective reasoning strategies
• Incurs overthinking (generating more tokens without improvement)

Key Contribution: Verifiable Process Supervision (VPS)

1. Belief Probing: Track model’s conditional probability of correct answer at each reasoning step
2. Segment-wise Progress: Compute progress measurements for discrete segments
3. Refined Feedback: Replace GRPO’s trajectory-level feedback with segment-wise rewards

Results

Across math and general-domain benchmarks:

Metric	Improvement
Math Accuracy	+2.6 points
Reasoning Length	-13.7%
General Domain Accuracy	+2.4 points
General Domain Length	-4%

Takeaways

• Process supervision can be model-free and verifiable
• Credit assignment at segment level is more efficient than trajectory level
• Reduces overthinking while improving accuracy
• Avoids costly Monte Carlo rollouts or auxiliary models