Safe Continual Reinforcement Learning under Nonstationarity via Adaptive Safety Constraints

TL;DR

This paper introduces LILAC+, an adaptive safety framework for continual reinforcement learning that dynamically adjusts safety constraints in response to environmental nonstationarity, improving safety and performance.

Contribution

The paper presents a novel framework combining three adaptive safety mechanisms for safe reinforcement learning in changing environments, addressing limitations of fixed constraints.

Findings

Adaptive safety constraints reduce safety violations under distribution shift.

LILAC+ maintains competitive task performance with improved safety.

Framework effectively handles both seen and unseen nonstationary conditions.

Abstract

Safe reinforcement learning in nonstationary environments requires safety mechanisms that adapt as environmental conditions change. Standard safe reinforcement learning methods often assume fixed constraints or stable environmental conditions, which can become inadequate under distribution shift. We propose LILAC+, a framework for safe continual reinforcement learning under nonstationarity that combines three adaptive safety mechanisms: context-based safety constraints, adaptation-speed constraints, and budget-to-state safety enforcement. Context-based constraints adjust safety requirements using inferred and predicted environmental context. Adaptation-speed constraints tighten safety requirements when the rate of environmental change exceeds the agent's ability to adapt safely. Budget-to-state enforcement converts cumulative safety requirements into local state-level control constraints that can be enforced at decision time. Together, these mechanisms provide a unified approach for proactive and reactive safety adaptation in continual reinforcement learning. We evaluate the framework in simulated driving environments under stationary, seen nonstationary, and unseen nonstationary conditions. The results show that adaptive safety constraints substantially reduce safety violations under distribution shift while maintaining competitive task performance compared with unconstrained and fixed-constraint baselines. These findings suggest that safe continual reinforcement learning requires adaptive constraint mechanisms that respond not only to current state information but also to predicted environmental context, adaptation demand, and remaining safety budget.