Safe Reinforcement Learning Using Robust Control Barrier Functions

TL;DR

This paper introduces a method combining reinforcement learning with a differentiable robust control barrier function layer to ensure safety and improve exploration, especially in safety-critical systems.

Contribution

It proposes a novel safety framework integrating control barrier functions into RL and a modular reward learning approach for enhanced safety and transferability.

Findings

Ensures safety during RL training.

Improves exploration efficiency.

Enables zero-shot transfer with modular reward learning.

Abstract

Reinforcement Learning (RL) has been shown to be effective in many scenarios. However, it typically requires the exploration of a sufficiently large number of state-action pairs, some of which may be unsafe. Consequently, its application to safety-critical systems remains a challenge. An increasingly common approach to address safety involves the addition of a safety layer that projects the RL actions onto a safe set of actions. In turn, a difficulty for such frameworks is how to effectively couple RL with the safety layer to improve the learning performance. In this paper, we frame safety as a differentiable robust-control-barrier-function layer in a model-based RL framework. Moreover, we also propose an approach to modularly learn the underlying reward-driven task, independent of safety constraints. We demonstrate that this approach both ensures safety and effectively guides exploration during training in a range of experiments, including zero-shot transfer when the reward is learned in a modular way.