SMAC: Score-Matched Actor-Critics for Robust Offline-to-Online Transfer

Paper: 2602.17632 Authors: Nathan S. de Lara, Florian Shkurti Categories: cs.LG, cs.AI

Abstract

This paper addresses a critical challenge in reinforcement learning: the performance collapse that occurs when fine-tuning offline-trained actor-critics with online value-based algorithms. The authors propose Score Matched Actor-Critic (SMAC), which hypothesizes that offline and online performance maxima are separated by low-performance valleys in the loss landscape. SMAC introduces a regularization technique that enforces a first-order derivative equality between the policy score function and the action-gradient of the Q-function during offline training. This approach creates smooth optimization paths between offline and online maxima, enabling seamless transfer to algorithms like Soft Actor-Critic (SAC) and TD3. Experimental results on D4RL benchmarks demonstrate perfect transfer success (6/6 tasks) with substantial regret reduction (34-58% in 4/6 environments).

Key Contributions

• Loss Landscape Analysis: Provides empirical evidence that offline and online maxima are separated by performance valleys, explaining why standard fine-tuning fails
• Score Matching Regularization: Introduces a novel regularization term that enforces $\nabla_a \log \pi(a|s) = \nabla_a Q(s,a)$ during offline training
• Smooth Transfer Guarantee: Demonstrates that SMAC finds offline solutions connected to online maxima via monotonically improving paths
• Strong Empirical Results: Achieves 100% successful transfer rate across D4RL tasks with significant regret reduction over baselines

Methodology

SMAC’s core innovation lies in its regularization approach during the offline training phase. Traditional offline RL methods optimize the actor and critic independently, leading to misalignment when transitioning to online learning. SMAC addresses this by adding a score-matching regularization term to the critic loss:

$\mathcal{L}_{\text{SMAC}} = \mathcal{L}_{\text{critic}} + \lambda \mathbb{E}_{(s,a) \sim \mathcal{D}} \left[\|\nabla_a Q(s,a) - \nabla_a \log \pi(a|s)\|^2\right]$

This regularization ensures that the Q-function’s action-gradient aligns with the policy’s score function. The theoretical motivation stems from the observation that in optimal actor-critic solutions, these gradients should be proportional. By enforcing this relationship during offline training, SMAC creates a Q-function that naturally supports gradient-based policy improvement in the online phase.

The algorithm maintains the standard actor-critic architecture but modifies the critic update to respect this constraint. During offline training, the policy is trained using standard policy gradient methods, while the Q-function is trained with both the Bellman error and the score-matching penalty. This dual objective shapes the loss landscape such that the offline optimum lies on a ridge connecting to better online optima, rather than in an isolated valley.

Experimental Results

The authors evaluate SMAC on six D4RL benchmark tasks spanning different domains and dataset qualities. The key findings include:

Transfer Success: SMAC achieves successful transfer to both SAC and TD3 in all 6/6 tested environments, with “successful” defined as maintaining or improving performance during the initial online training phase. Baseline methods like IQL, CQL, and TD3+BC frequently exhibit immediate performance drops of 20-40% upon online fine-tuning.

Regret Reduction: In 4 out of 6 environments (halfcheetah-medium-expert, hopper-medium, walker2d-medium-replay, and antmaze-medium-play), SMAC reduces cumulative regret by 34-58% compared to the best performing baseline. The regret metric measures the area between the optimal policy’s performance and the agent’s performance during online learning.

Convergence Speed: SMAC not only avoids initial performance drops but also converges to higher final performance faster than baselines. The monotonic improvement property means that every gradient step during online fine-tuning leads to measurable reward increases, eliminating the recovery phase required by other methods.

Ablation Studies: The authors demonstrate that the score-matching coefficient $\lambda$ is critical—too low fails to prevent valleys, while too high over-constrains the Q-function. The optimal range appears to be $\lambda \in [0.1, 1.0]$ depending on the environment.

Implications and Future Directions

SMAC represents a significant advance in offline-to-online RL by addressing the fundamental geometric properties of the loss landscape rather than just improving offline performance. This perspective opens several research directions:

Theoretical Understanding: While the paper provides empirical evidence for the valley hypothesis, a formal characterization of when and why these valleys form would strengthen the theoretical foundation. Understanding the relationship between dataset coverage, function approximation, and landscape geometry could lead to more principled algorithm design.

Scalability: The score-matching regularization requires computing action-gradients of both the Q-function and policy, which adds computational overhead. Investigating efficient approximations or sampling-based methods could make SMAC more practical for high-dimensional action spaces.

Beyond Actor-Critic: The core insight about aligning gradients during offline training might extend to other RL paradigms. Model-based methods, for instance, could benefit from similar regularization to ensure smooth transfer when switching from model-based planning to model-free fine-tuning.

Multi-Task Transfer: SMAC’s success in single-task offline-to-online transfer suggests potential for multi-task scenarios where an agent must transfer knowledge across different tasks or domains. The score-matching principle could help maintain performance across task boundaries.

Takeaways

1. Performance drops during offline-to-online transfer stem from loss landscape geometry, specifically valleys separating offline and online maxima, rather than just poor offline training.

2. Score-matching regularization creates smooth optimization paths by enforcing $\nabla_a Q(s,a) \propto \nabla_a \log \pi(a|s)$ during offline training, enabling gradient-based fine-tuning without performance collapse.

3. SMAC achieves 100% transfer success rate across D4RL benchmarks with 34-58% regret reduction in most environments, demonstrating practical effectiveness.

4. The approach is algorithm-agnostic for online fine-tuning, successfully transferring to both SAC and TD3 without modification.

5. Geometric considerations in loss landscapes should be a primary concern when designing offline RL algorithms intended for online deployment, not just an afterthought.