Safety Filtering While Training: Improving the Performance and Sample Efficiency of Reinforcement Learning Agents

TL;DR

This paper explores training reinforcement learning controllers with safety filters integrated during training, leading to improved performance, safety, and sample efficiency, demonstrated through drone experiments and comprehensive analysis.

Contribution

It introduces training modifications for safety filters in RL, enhancing performance and robustness compared to traditional evaluation-only application.

Findings

Training with safety filters improves RL performance.

Modifications enhance sample efficiency and safety.

Guidelines for practitioners on safety filter integration.

Abstract

Reinforcement learning (RL) controllers are flexible and performant but rarely guarantee safety. Safety filters impart hard safety guarantees to RL controllers while maintaining flexibility. However, safety filters can cause undesired behaviours due to the separation between the controller and the safety filter, often degrading performance and robustness. In this paper, we analyze several modifications to incorporating the safety filter in training RL controllers rather than solely applying it during evaluation. The modifications allow the RL controller to learn to account for the safety filter, improving performance. This paper presents a comprehensive analysis of training RL with safety filters, featuring simulated and real-world experiments with a Crazyflie 2.0 drone. We examine how various training modifications and hyperparameters impact performance, sample efficiency, safety, and chattering. Our findings serve as a guide for practitioners and researchers focused on safety filters and safe RL.