Safe Reinforcement Learning Using Black-Box Reachability Analysis

TL;DR

This paper introduces a novel safety layer for reinforcement learning that uses black-box reachability analysis to ensure safety constraints are respected during robot motion planning, even with unknown models.

Contribution

The paper presents a new Black-box Reachability-based Safety Layer (BRSL) that combines data-driven reachability, online neural network ensemble prediction, and differentiable collision checks for safe RL.

Findings

BRSL outperforms existing safe RL methods in simulation tests

Effective safety guarantees are achieved without sacrificing performance

Applicable to various robotic platforms in uncertain environments

Abstract

Reinforcement learning (RL) is capable of sophisticated motion planning and control for robots in uncertain environments. However, state-of-the-art deep RL approaches typically lack safety guarantees, especially when the robot and environment models are unknown. To justify widespread deployment, robots must respect safety constraints without sacrificing performance. Thus, we propose a Black-box Reachability-based Safety Layer (BRSL) with three main components: (1) data-driven reachability analysis for a black-box robot model, (2) a trajectory rollout planner that predicts future actions and observations using an ensemble of neural networks trained online, and (3) a differentiable polytope collision check between the reachable set and obstacles that enables correcting unsafe actions. In simulation, BRSL outperforms other state-of-the-art safe RL methods on a Turtlebot 3, a quadrotor, a trajectory-tracking point mass, and a hexarotor in wind with an unsafe set adjacent to the area of highest reward.