Dyna-Style Safety Augmented Reinforcement Learning: Staying Safe in the Face of Uncertainty

TL;DR

Dyna-SAuR is a reinforcement learning algorithm that learns safety filters and control policies using uncertainty-aware models, significantly reducing failures in high-dimensional systems during training.

Contribution

It introduces a scalable safety filter and control policy learning method that requires minimal domain knowledge and improves safety in RL training.

Findings

Reduces failures by 100x on CartPole and MuJoCo Walker tasks.

Learns safety filters and policies with minimal domain knowledge.

Effectively manages uncertainty to expand safe state sets.

Abstract

Safety remains an open problem in reinforcement learning (RL), especially during training. While safety filters are promising to address safe exploration, they are generally poorly suited for high-dimensional systems with unknown dynamics. We propose Dyna-style Safety Augmented Reinforcement Learning (Dyna-SAuR), a novel algorithm that learns both a scalable safety filter and a control policy using a learned uncertainty-aware dynamics model, while requiring minimal domain knowledge. The filter avoids failures and high uncertainty regions. Thus, better models expand the set of safe and certain states, reducing filter conservatism. We present the effectiveness of Dyna-SAuR on goal-reaching CartPole as well as MuJoCo Walker, reducing failures compared to state-of-the-art methods by 2 orders of magnitude.