Implicit Safe Set Algorithm for Provably Safe Reinforcement Learning

TL;DR

This paper introduces a model-free safe control algorithm for reinforcement learning that guarantees safety during training by using a black-box dynamic function, addressing a key challenge in deploying DRL in real-world safety-critical applications.

Contribution

The paper proposes the implicit safe set algorithm, a novel method that ensures provable safety for DRL agents without requiring explicit system models, using only black-box queries.

Findings

Guarantees finite-time convergence to the safe set.

Achieves zero safety violations on Safety Gym benchmark.

Scales efficiently to high-dimensional systems.

Abstract

Deep reinforcement learning (DRL) has demonstrated remarkable performance in many continuous control tasks. However, a significant obstacle to the real-world application of DRL is the lack of safety guarantees. Although DRL agents can satisfy system safety in expectation through reward shaping, designing agents to consistently meet hard constraints (e.g., safety specifications) at every time step remains a formidable challenge. In contrast, existing work in the field of safe control provides guarantees on persistent satisfaction of hard safety constraints. However, these methods require explicit analytical system dynamics models to synthesize safe control, which are typically inaccessible in DRL settings. In this paper, we present a model-free safe control algorithm, the implicit safe set algorithm, for synthesizing safeguards for DRL agents that ensure provable safety throughout training. The proposed algorithm synthesizes a safety index (barrier certificate) and a subsequent safe control law solely by querying a black-box dynamic function (e.g., a digital twin simulator). Moreover, we theoretically prove that the implicit safe set algorithm guarantees finite time convergence to the safe set and forward invariance for both continuous-time and discrete-time systems. We validate the proposed algorithm on the state-of-the-art Safety Gym benchmark, where it achieves zero safety violations while gaining $95\% \pm 9\%$ cumulative reward compared to state-of-the-art safe DRL methods. Furthermore, the resulting algorithm scales well to high-dimensional systems with parallel computing.