Safe Continual Reinforcement Learning in Non-stationary Environments

TL;DR

This paper investigates safe continual reinforcement learning in non-stationary environments, highlighting the challenges of balancing safety and adaptation, and evaluating existing methods through new benchmarks.

Contribution

It introduces three benchmark environments for safe continual RL and analyzes the trade-offs between safety and catastrophic forgetting.

Findings

Existing methods struggle to maintain safety while adapting to changing dynamics.

Regularization strategies can partially mitigate safety-forgetting trade-offs.

Fundamental tension exists between safety constraints and continual adaptation in non-stationary settings.

Abstract

Reinforcement learning (RL) offers a compelling data-driven paradigm for synthesizing controllers for complex systems when accurate physical models are unavailable; however, most existing control-oriented RL methods assume stationarity and, therefore, struggle in real-world non-stationary deployments where system dynamics and operating conditions can change unexpectedly. Moreover, RL controllers acting in physical environments must satisfy safety constraints throughout their learning and execution phases, rendering transient violations during adaptation unacceptable. Although continual RL and safe RL have each addressed non-stationarity and safety, respectively, their intersection remains comparatively unexplored, motivating the study of safe continual RL algorithms that can adapt over the system's lifetime while preserving safety. In this work, we systematically investigate safe continual reinforcement learning by introducing three benchmark environments that capture safety-critical continual adaptation and by evaluating representative approaches from safe RL, continual RL, and their combinations. Our empirical results reveal a fundamental tension between maintaining safety constraints and preventing catastrophic forgetting under non-stationary dynamics, with existing methods generally failing to achieve both objectives simultaneously. To address this shortcoming, we examine regularization-based strategies that partially mitigate this trade-off and characterize their benefits and limitations. Finally, we outline key open challenges and research directions toward developing safe, resilient learning-based controllers capable of sustained autonomous operation in changing environments.