Safe Continual Reinforcement Learning in Non-stationary Environments

Paper: 2604.19737 Authors: Austin Coursey, Abel Diaz-Gonzalez, Marcos Quinones-Grueiro, Gautam Biswas (Vanderbilt) Categories: cs.LG, cs.RO, cs.AI

Problem

While RL offers data-driven paradigm for complex systems:

• Most methods assume stationarity - struggle in real-world non-stationary deployments
• RL controllers must satisfy safety constraints throughout learning and execution
• Intersection of safe RL and continual RL remains unexplored

Key Insight: Fundamental Tension

Empirical results reveal a fundamental tension:

• Maintaining safety constraints
• Preventing catastrophic forgetting under non-stationary dynamics

Existing methods generally fail to achieve both objectives simultaneously.

Benchmark Environments

Three robotic benchmarks for safe continual RL:

1. Damaged Half-Cheetah: Running with velocity constraints
2. Damaged Multi-leg Ant Robot: Crawling with velocity constraints
3. Robotic Arm Coffee Mug: Avoiding spills during everyday tasks

Proposed Algorithms

Two candidate algorithms:

• Safe EWC (Elastic Weight Consolidation)
• CF EWC (Cost-Fisher EWC)

Regularization-based strategies that partially mitigate the safety-forgetting trade-off.

Takeaways

• Real-world deployment requires both adaptation and knowledge retention
• Safety constraints add complexity to continual learning
• Need new algorithms that balance both objectives