From Cumulative Constraints to Adaptive Runtime Safety Control for Nonstationary Reinforcement Learning

TL;DR

This paper introduces CPSS, a runtime safety mechanism for reinforcement learning that adaptively converts cumulative safety budgets into local, state-level safety constraints, improving safety guarantees in nonstationary environments.

Contribution

The paper proposes a novel adaptive safety shield, CPSS, that dynamically adjusts safety thresholds based on contextual signals to ensure safety in nonstationary reinforcement learning.

Findings

CPSS reduces safety violations in highway merging scenarios.

CPSS enforces safety constraints while maintaining policy performance.

CPSS adapts safety thresholds online based on environment demands.

Abstract

Safety in reinforcement learning is often specified through cumulative cost constraints, but these trajectory-level guarantees do not directly prevent unsafe individual decisions, especially under nonstationarity. In continual and nonstationary settings, the difficulty is amplified because the risk associated with the same action can vary across contexts, while a fixed state-level threshold may be either too conservative or too weak. We propose Constraint Projection Safety Shield (CPSS), a runtime mechanism that converts a cumulative safety budget into adaptive state-level control constraints during execution. CPSS tracks the remaining safety budget, projects it into a time-varying admissible risk threshold, and filters policy actions whose predicted safety cost exceeds the active threshold. The threshold is adjusted online using contextual signals so that enforcement becomes stricter in more demanding or rapidly changing regimes and less restrictive when the available safety budget is sufficient. We analyze the resulting shielded policy and show that the mechanism guarantees per-state threshold satisfaction for executed actions, induces finite-horizon cumulative cost bounds, and yields a performance degradation bound in terms of intervention frequency and per-step reward distortion. We evaluate CPSS in nonstationary highway merging scenarios using highway-env. Across multiple seeds, CPSS substantially reduces proximity-based safety violations and increases separation margins while intervening selectively rather than dominating the learned policy. These results support adaptive budget-to-threshold projection as a practical way to transform cumulative safety specifications into effective local safety control for continual reinforcement learning systems.