FASTER: Value-Guided Sampling for Fast Reinforcement Learning

Paper: 2604.19730 Authors: Perry Dong, Alexander Swerdlow, Dorsa Sadigh, Chelsea Finn (Stanford) Categories: cs.LG, cs.RO, cs.AI

Problem

Most performant RL algorithms use test-time scaling methods like sampling multiple action candidates and selecting the best - but these are prohibitively expensive.

Key Insight

Model the denoising of multiple action candidates + selecting the best as a Markov Decision Process (MDP):

• Goal: progressively filter action candidates before denoising is complete
• Learn policy and value function in denoising space
• Predict downstream value of action candidates

FASTER Method

A method for getting benefits of sampling-based test-time scaling without the computational cost:

1. Early Filtering: Trace performance gain of action samples back to earlier denoising stages
2. Value Prediction: Learn to predict which candidates will perform best
3. Lightweight: Can be plugged into existing generative RL algorithms

Results

Across challenging long-horizon manipulation tasks (online and batch-online RL):

Setting	Result
Task Performance	Consistent improvement
Training/Inference Compute	Substantially reduced
VLA Application	Same performance, less compute

Takeaways

• Sampling-based scaling benefits achievable without expensive sampling
• MDP formulation enables principled early filtering
• Plug-and-play design works with diverse RL algorithms