TOPReward: Token Probabilities as Hidden Zero-Shot Rewards for Robotics

Paper: 2602.19313 Authors: Shirui Chen, Cole Harrison, Ying-Chun Lee, Angela Jin Yang, Zhongzheng Ren, Lillian J. Ratliff, Jiafei Duan, Dieter Fox, Ranjay Krishna Categories: cs.RO, cs.AI, cs.LG

Abstract

Vision-Language-Action (VLA) models have made significant strides in pretraining, but their application to reinforcement learning faces critical challenges: low sample efficiency and sparse reward signals in real-world robotics. The core issue lies in developing generalizable process reward models that can provide fine-grained feedback across diverse tasks. TOPReward addresses this by introducing a probabilistically grounded temporal value function that taps into the latent world knowledge embedded in pretrained video Vision-Language Models (VLMs). Rather than prompting VLMs to directly output numerical progress estimates—a method prone to misrepresentation—TOPReward extracts task progress information directly from the model’s internal token logits. Evaluated zero-shot across over 130 distinct real-world robotic tasks spanning multiple platforms (Franka, YAM, SO-100/101), TOPReward achieves a remarkable 0.947 mean Value-Order Correlation (VOC) on Qwen3-VL, vastly outperforming the state-of-the-art GVL baseline which struggles with near-zero correlation on the same model.

Key Contributions

• Introduction of TOPReward, a novel temporal value function that extracts task progress from VLM token logits rather than prompted numerical outputs
• Probabilistic grounding that leverages pretrained video VLMs’ latent world knowledge for zero-shot generalization
• Comprehensive zero-shot evaluation across 130+ real-world robotic tasks on multiple platforms, achieving 0.947 mean VOC
• Demonstration of TOPReward’s versatility for downstream applications including success detection and reward-aligned behavior cloning
• Significant performance improvement over state-of-the-art methods, with GVL baseline achieving near-zero correlation on the same open-source model

Technical Methodology

The fundamental innovation of TOPReward lies in its approach to extracting reward signals from VLMs. Traditional methods prompt VLMs to generate explicit numerical progress values, but this approach suffers from a critical flaw: language models are notoriously poor at representing precise numerical quantities through their text generation process. TOPReward circumvents this limitation by directly accessing the model’s internal token probability distributions.

The method operates on video sequences of robotic task execution. Given a video trajectory, TOPReward queries the VLM with task-specific prompts and analyzes the probability distribution over tokens in the model’s vocabulary. Rather than sampling from this distribution to generate text, TOPReward extracts the logits (pre-softmax values) associated with tokens that indicate task progress or completion. This probabilistic signal is inherently more reliable than generated text because it reflects the model’s internal confidence about task state without the distortion introduced by the sampling and decoding process.

The temporal value function $V(s_t)$ at state $s_t$ is computed by aggregating token probabilities across relevant progress indicators. The model can be formulated as:

$V(s_t) = f(\text{logits}(\text{VLM}(v_{1:t}, p)))$

where $v_{1:t}$ represents the video frames up to time $t$, $p$ is the task prompt, and $f$ is an aggregation function over the relevant token logits. This formulation provides a continuous, differentiable signal that can be used for reinforcement learning optimization.

Experimental Results and Analysis

The experimental validation of TOPReward is extensive and compelling. Across 130+ distinct real-world robotic tasks, the method demonstrates exceptional zero-shot generalization capabilities. The 0.947 mean Value-Order Correlation (VOC) on Qwen3-VL represents a dramatic improvement over existing approaches. VOC measures how well the predicted value function preserves the ordering of states according to their true progress toward task completion—a critical property for effective reward shaping in RL.

The comparison with the GVL baseline is particularly striking. While GVL represents the state-of-the-art in prompted VLM-based reward modeling, it achieves near-zero correlation on the same open-source Qwen3-VL model. This performance gap highlights the fundamental advantage of TOPReward’s approach: by accessing internal model representations rather than relying on generated text, it captures more reliable and nuanced progress signals.

The evaluation spans multiple robot platforms including Franka robotic arms, YAM mobile manipulators, and SO-100/101 systems, demonstrating that TOPReward’s effectiveness is not platform-specific but rather generalizes across different embodiments and task types. This cross-platform generalization is crucial for practical deployment in diverse robotic applications.

Implications for Robotic Learning

TOPReward addresses a fundamental bottleneck in robotic reinforcement learning: the reward specification problem. In real-world settings, defining reward functions that are both informative and generalizable is notoriously difficult. Sparse rewards (e.g., binary success/failure) provide insufficient learning signal, while hand-crafted dense rewards require extensive domain expertise and often fail to generalize to new tasks.

By leveraging the world knowledge embedded in large-scale pretrained VLMs, TOPReward provides a path toward more sample-efficient robotic learning. The fine-grained progress signals it generates can guide policy optimization more effectively than sparse rewards, potentially reducing the number of real-world interactions required for learning. This is particularly valuable in robotics where data collection is expensive and time-consuming.

The versatility demonstrated through downstream applications—success detection and reward-aligned behavior cloning—suggests that TOPReward can serve as a foundational component in robotic learning systems. Success detection enables automated evaluation and curriculum learning, while reward-aligned behavior cloning allows for more effective imitation learning by focusing on high-reward trajectories.

Takeaways

1. TOPReward introduces a paradigm shift in VLM-based reward modeling by extracting signals from token logits rather than generated text, achieving 0.947 VOC across 130+ real-world robotic tasks.

2. The method’s probabilistic grounding enables zero-shot generalization across diverse tasks and robot platforms without requiring task-specific training or fine-tuning.

3. By dramatically outperforming state-of-the-art prompted approaches (GVL baseline achieves near-zero correlation on the same model), TOPReward demonstrates the superiority of accessing internal model representations.

4. The approach addresses critical challenges in robotic RL—sparse rewards and low sample efficiency—by providing fine-grained, generalizable progress signals derived from pretrained VLM world knowledge.

5. TOPReward’s versatility for downstream applications including success detection and reward-aligned behavior cloning positions it as a foundational tool for next-generation robotic learning systems.