Reachability-based Trajectory Safeguard (RTS): A Safe and Fast Reinforcement Learning Safety Layer for Continuous Control

TL;DR

This paper introduces RTS, a safety layer for reinforcement learning in continuous control that uses reachability analysis to ensure safety during training and operation, demonstrated on complex robot models including a quadrotor.

Contribution

The paper presents a novel reachability-based safety layer that precomputes reachable sets to guarantee safety during RL training and deployment in real-world robots.

Findings

RTS effectively prevents unsafe actions in simulated robot models.

The method outperforms existing safe motion planning techniques.

RTS is applicable to high-dimensional nonlinear systems like quadrotors.

Abstract

Reinforcement Learning (RL) algorithms have achieved remarkable performance in decision making and control tasks due to their ability to reason about long-term, cumulative reward using trial and error. However, during RL training, applying this trial-and-error approach to real-world robots operating in safety critical environment may lead to collisions. To address this challenge, this paper proposes a Reachability-based Trajectory Safeguard (RTS), which leverages reachability analysis to ensure safety during training and operation. Given a known (but uncertain) model of a robot, RTS precomputes a Forward Reachable Set of the robot tracking a continuum of parameterized trajectories. At runtime, the RL agent selects from this continuum in a receding-horizon way to control the robot; the FRS is used to identify if the agent's choice is safe or not, and to adjust unsafe choices. The efficacy of this method is illustrated on three nonlinear robot models, including a 12-D quadrotor drone, in simulation and in comparison with state-of-the-art safe motion planning methods.