Anomalous State Sequence Modeling to Enhance Safety in Reinforcement Learning

TL;DR

This paper introduces AnoSeqs, a novel safe reinforcement learning method that uses anomaly detection on state sequences to improve safety in high-stakes environments like self-driving cars.

Contribution

It presents a two-stage approach combining offline safe sequence collection and anomaly detection to guide risk-averse RL policy training in safety-critical settings.

Findings

Successfully learned safer policies in multiple benchmarks

Effective anomaly detection provided supervisory signals for safety

Improved safety performance in self-driving car simulations

Abstract

The deployment of artificial intelligence (AI) in decision-making applications requires ensuring an appropriate level of safety and reliability, particularly in changing environments that contain a large number of unknown observations. To address this challenge, we propose a novel safe reinforcement learning (RL) approach that utilizes an anomalous state sequence to enhance RL safety. Our proposed solution Safe Reinforcement Learning with Anomalous State Sequences (AnoSeqs) consists of two stages. First, we train an agent in a non-safety-critical offline 'source' environment to collect safe state sequences. Next, we use these safe sequences to build an anomaly detection model that can detect potentially unsafe state sequences in a 'target' safety-critical environment where failures can have high costs. The estimated risk from the anomaly detection model is utilized to train a risk-averse RL policy in the target environment; this involves adjusting the reward function to penalize the agent for visiting anomalous states deemed unsafe by our anomaly model. In experiments on multiple safety-critical benchmarking environments including self-driving cars, our solution approach successfully learns safer policies and proves that sequential anomaly detection can provide an effective supervisory signal for training safety-aware RL agents