Safe Reinforcement Learning using Data-Driven Predictive Control

TL;DR

This paper introduces a data-driven safety layer for reinforcement learning that ensures safety during decision-making by verifying actions through reachability analysis, improving safety in robotics control tasks.

Contribution

It presents a novel safety layer using data-driven predictive control to verify and replace unsafe actions in RL, enhancing safety guarantees during training and deployment.

Findings

Outperforms state-of-the-art safe RL methods in robotics navigation tasks.

Successfully applied to Turtlebot 3 and quadrotor simulations.

Ensures safety without significantly compromising performance.

Abstract

Reinforcement learning (RL) algorithms can achieve state-of-the-art performance in decision-making and continuous control tasks. However, applying RL algorithms on safety-critical systems still needs to be well justified due to the exploration nature of many RL algorithms, especially when the model of the robot and the environment are unknown. To address this challenge, we propose a data-driven safety layer that acts as a filter for unsafe actions. The safety layer uses a data-driven predictive controller to enforce safety guarantees for RL policies during training and after deployment. The RL agent proposes an action that is verified by computing the data-driven reachability analysis. If there is an intersection between the reachable set of the robot using the proposed action, we call the data-driven predictive controller to find the closest safe action to the proposed unsafe action. The safety layer penalizes the RL agent if the proposed action is unsafe and replaces it with the closest safe one. In the simulation, we show that our method outperforms state-of-the-art safe RL methods on the robotics navigation problem for a Turtlebot 3 in Gazebo and a quadrotor in Unreal Engine 4 (UE4).