SOLE-R1: Teaching Robots Through Video Reasoning Alone

Paper: 2603.28730 Authors: Philip Schroeder, Thomas Weng, Karl Schmeckpeper, Eric Rosen, Stephen Hart, Ondrej Biza Categories: cs.RO, cs.CL, cs.CV

The Gap

Vision-language models (VLMs) like GPT-4V and Gemini have reached impressive capabilities on static image tasks, and researchers naturally want to use them to supervise robot learning. The standard approach: use VLMs as reward evaluators in reinforcement learning (RL) — the robot tries actions, the VLM judges success, the robot learns from that feedback.

But there’s a critical failure mode: when robots operate in partially observable environments (can’t see everything) or face distribution shift (training differs from deployment), today’s VLMs make perceptual errors. Worse, RL policies are optimization machines — they exploit these errors rather than solve the actual task. The robot learns to “hack” the reward signal instead of learning the skill.

Prior work tried patching this with better prompting, ensemble methods, or human-in-the-loop corrections. None fundamentally solved the core issue: existing VLMs weren’t designed to be robust reward functions for online RL.

Problem: VLMs as reward evaluators
   |
   v
Assumption: Perceptual errors stem from lack of temporal reasoning
   |
   v
Method: Train video-language model with spatiotemporal CoT + dense progress signals
   |
   v
Evidence: Zero-shot RL on 24 unseen tasks, outperforms GPT-5/Gemini-3-Pro
   |
   v
Conclusion: Purpose-built video reasoning enables robust reward-only RL

The Increment

One sentence: Before, using VLMs for robot RL required ground-truth rewards or demonstrations to prevent reward hacking; after, a specialized video reasoning model can serve as the sole supervision signal for learning from scratch.

Core Mechanism

SOLE-R1 is a video-language model that watches robot video streams and produces per-timestep progress estimates (0-1 scores) toward a natural language goal. Unlike standard VLMs that process single frames or short clips, SOLE-R1 performs spatiotemporal chain-of-thought reasoning: at each timestep, it generates a textual reasoning trace describing what’s happening spatially in the scene and how that relates to task progress over time, then outputs a continuous progress score.

The training pipeline has three stages. First, they synthesize a large-scale dataset of robot trajectories with temporally grounded reasoning traces — not just “task succeeded/failed” labels, but detailed per-frame explanations of why progress is 0.3 vs 0.7. Second, they combine this synthetic data with foundational spatial reasoning (from image-text pairs) and multi-frame temporal reasoning (from video captioning). Third, they train with a hybrid approach: supervised fine-tuning on the reasoning traces, plus RL from verifiable rewards (using ground-truth task state when available during training only).

At deployment, SOLE-R1 receives only raw video pixels and a language goal. No ground-truth state, no success detectors, no demonstrations. It outputs dense progress estimates that serve directly as RL rewards. The robot’s policy network learns purely from these video-derived rewards through standard online RL (they use PPO).

Video stream (t=0,1,2,...)  +  Language goal
         |                           |
         v                           v
    [SOLE-R1 Video Encoder]--->[Reasoning Module]
         |                           |
         |---> Spatial attention     |
         |---> Temporal attention    |
         |                           v
         |                    CoT reasoning trace
         |                    ("gripper approaching cup...")
         |                           |
         v                           v
    [Progress Estimator]<-----------+
         |
         v
    Progress score (0.0 to 1.0)
         |
         v
    RL reward signal ----> Policy learning

Think of SOLE-R1 as a sports commentator who’s also a coach. A regular commentator (standard VLM) watches the game and says “good play” or “bad play” at the end. But SOLE-R1 is like a commentator who narrates every moment — “the player is positioning for a shot, now pivoting left, weight shifting forward” — and simultaneously tracks a progress meter: 20% toward scoring, now 40%, now 60%. The robot (player) doesn’t need to know the rules or see the scoreboard; it just learns to maximize that progress meter. The key is that the commentator understands both the spatial details (where is the ball?) and temporal dynamics (is this motion building toward a goal?), so it can’t be fooled by superficial tricks like moving randomly near the target.

Key Concepts

• Spatiotemporal Chain-of-Thought: Standard chain-of-thought prompting generates reasoning steps for a single decision. Spatiotemporal CoT extends this to video: at each frame, the model generates text describing both what’s spatially present (“gripper 5cm from cup handle”) and how the temporal sequence relates to progress (“approaching motion continued from previous frame, distance decreasing”). This grounds the reasoning in both space and time, making it harder to exploit. Concrete example: if a robot randomly waves near a cup, spatial CoT might say “gripper near cup” (exploitable), but spatiotemporal CoT says “gripper near cup but motion is erratic, not consistent with grasping trajectory” (robust).

• Dense Progress Supervision: Traditional RL rewards are sparse (0 until success, then 1) or shaped by hand. Dense progress supervision means every timestep gets a continuous score (0.0 to 1.0) representing partial completion. The insight: this is learnable from video if you have enough examples. They synthesize training data by taking successful trajectories, segmenting them, and labeling each segment’s progress based on how far through the task it is. A robot halfway through “pick up cup” gets ~0.5, not 0. This density prevents the policy from getting stuck in local optima and provides richer learning signal.

• Reward Hacking vs Robust Rewards: Reward hacking occurs when a policy exploits flaws in the reward function rather than solving the intended task. Example: if the reward is “gripper close to cup,” the robot learns to hover near the cup without grasping. Robust rewards resist this by capturing task semantics, not just surface features. SOLE-R1 achieves robustness through temporal reasoning — it checks whether the sequence of actions makes sense for the task, not just whether individual frames look right. The RL-from-verifiable-rewards training stage explicitly teaches the model to resist common hacking strategies by showing it failure modes during training.

Framework Shift

Before (mainstream approach):        After (this paper):

VLM (image-based)                    SOLE-R1 (video-based)
      |                                     |
      v                                     v
  [Frame] ---> "success?"            [Video stream] ---> CoT reasoning
      |            |                        |                  |
      v            v                        v                  v
  Sparse reward (0/1)              Dense progress (0.0-1.0)
      |                                     |
      v                                     v
  Policy exploits errors           Policy learns robust behavior
      |                                     |
      v                                     v
  Needs ground-truth backup        Works with video alone

One sentence: From frame-level binary judgments to video-level continuous reasoning, the core shift is treating reward generation as a temporal reasoning problem rather than a classification problem.

Expert Assessment

Problem choice: This is a real gap. The robotics community has been trying to use foundation models for reward specification for 2-3 years, and reward hacking is a documented, persistent failure mode. The problem sits at a critical juncture: foundation models are powerful enough to understand tasks but not robust enough to supervise learning. Solving this unlocks a lot of downstream applications.

Method maturity: The approach is well-engineered but not particularly novel conceptually. Spatiotemporal reasoning for video understanding is established; applying it to reward modeling is the increment. The synthesis pipeline for generating reasoning traces is clever but labor-intensive (requires task simulators and heuristics). The hybrid training (SFT + RL from verifiable rewards) is sensible but raises questions: how much does each component contribute? Ablations would strengthen the story. One overlooked simpler approach: could you just fine-tune existing video models (e.g., VideoLLaMA) on progress-labeled trajectories without the full CoT apparatus?

Experimental integrity: Baselines are strong (GPT-5, Gemini-3-Pro) and the comparison is fair. The zero-shot setting is genuinely impressive — no task-specific tuning is a high bar. However, the real-robot experiments are limited (one environment, qualitative results). The claim of “substantially outperforms” is supported in simulation but needs more real-world validation. One red flag: they don’t report wall-clock training time or compute costs for SOLE-R1 training, which matters for reproducibility. The reward hacking analysis (Section 4.3) is honest and shows failure modes, which builds trust.

Writing quality: The paper is clearly written overall, but Section 3.2 (training pipeline) is dense and would benefit from a figure showing the data flow. The related work section undersells prior work on video-based reward learning (e.g., AVID, R3M) — acknowledging these would position the contribution better. The conclusion oversells slightly (“enables zero-shot online RL”) when the method still requires substantial offline training on synthetic data.

Verdict: weak accept — Solid execution on an important problem with convincing simulation results, but needs stronger real-world validation and clearer ablations to be a strong accept.

Takeaways

For practitioners building robot learning systems: The key transferable idea is treating reward modeling as a video reasoning problem, not a frame classification problem. If you’re using VLMs for evaluation, add temporal consistency checks — does the sequence of observations make sense for the task? The spatiotemporal CoT pattern (describe what’s happening + how it relates to progress) is a concrete prompt engineering technique you can apply immediately.

For ML researchers: The hybrid training approach (SFT on reasoning traces + RL from verifiable rewards) is a template for training robust evaluators in any domain where you can synthesize data and have occasional access to ground truth. The insight: use RL during training to explicitly teach the model to resist exploitation, even if you won’t have ground truth at deployment.

For anyone working with foundation models: The paper demonstrates that purpose-built models can outperform larger general-purpose models (GPT-5, Gemini-3-Pro) on specialized tasks. The lesson: if you have a critical application, the investment in task-specific training data and model design pays off. Don’t assume the biggest model is always the best tool.