Safe Reinforcement Learning via Hierarchical Adaptive Chance-Constraint Safeguards

TL;DR

This paper introduces ACS, a model-free safe reinforcement learning algorithm that uses adaptive chance constraints to ensure safety during exploration and after convergence, balancing safety and optimality effectively.

Contribution

The paper proposes a novel adaptive chance-constrained safety safeguard for RL that guarantees safety with minimal violations and maintains high performance.

Findings

Achieves nearly zero safety violations in experiments

Improves reward by 23.8% while ensuring safety

Demonstrates robustness and fast response in real-world tasks

Abstract

Ensuring safety in Reinforcement Learning (RL), typically framed as a Constrained Markov Decision Process (CMDP), is crucial for real-world exploration applications. Current approaches in handling CMDP struggle to balance optimality and feasibility, as direct optimization methods cannot ensure state-wise in-training safety, and projection-based methods correct actions inefficiently through lengthy iterations. To address these challenges, we propose Adaptive Chance-constrained Safeguards (ACS), an adaptive, model-free safe RL algorithm using the safety recovery rate as a surrogate chance constraint to iteratively ensure safety during exploration and after achieving convergence. Theoretical analysis indicates that the relaxed probabilistic constraint sufficiently guarantees forward invariance to the safe set. And extensive experiments conducted on both simulated and real-world safety-critical tasks demonstrate its effectiveness in enforcing safety (nearly zero-violation) while preserving optimality (+23.8%), robustness, and fast response in stochastic real-world settings.