Dynamic Shielding for Reinforcement Learning in Black-Box Environments

TL;DR

This paper introduces dynamic shielding, a novel method for safe reinforcement learning in black-box environments that constructs approximate models in real-time to prevent unsafe actions without prior system knowledge.

Contribution

It extends model-based safe RL by integrating automata learning to dynamically construct shields, enabling safety guarantees without prior system information.

Findings

Significantly reduces undesired events during training.

Constructs approximate system models in parallel with RL.

Prevents unsafe actions before they occur.

Abstract

It is challenging to use reinforcement learning (RL) in cyber-physical systems due to the lack of safety guarantees during learning. Although there have been various proposals to reduce undesired behaviors during learning, most of these techniques require prior system knowledge, and their applicability is limited. This paper aims to reduce undesired behaviors during learning without requiring any prior system knowledge. We propose dynamic shielding: an extension of a model-based safe RL technique called shielding using automata learning. The dynamic shielding technique constructs an approximate system model in parallel with RL using a variant of the RPNI algorithm and suppresses undesired explorations due to the shield constructed from the learned model. Through this combination, potentially unsafe actions can be foreseen before the agent experiences them. Experiments show that our dynamic shield significantly decreases the number of undesired events during training.