Dynamic Model Predictive Shielding for Provably Safe Reinforcement Learning

TL;DR

This paper presents Dynamic Model Predictive Shielding (DMPS), a novel approach that enhances safe reinforcement learning by dynamically integrating a local planner with neural policies, leading to improved safety and performance.

Contribution

DMPS introduces a dynamic planning method that jointly optimizes safety and reward, reducing conservative shielding and enabling policies to learn effectively in complex environments.

Findings

DMPS achieves higher rewards than state-of-the-art baselines.

DMPS reduces shield interventions after training.

Safety during and after training is guaranteed with bounded regret.

Abstract

Among approaches for provably safe reinforcement learning, Model Predictive Shielding (MPS) has proven effective at complex tasks in continuous, high-dimensional state spaces, by leveraging a backup policy to ensure safety when the learned policy attempts to take risky actions. However, while MPS can ensure safety both during and after training, it often hinders task progress due to the conservative and task-oblivious nature of backup policies. This paper introduces Dynamic Model Predictive Shielding (DMPS), which optimizes reinforcement learning objectives while maintaining provable safety. DMPS employs a local planner to dynamically select safe recovery actions that maximize both short-term progress as well as long-term rewards. Crucially, the planner and the neural policy play a synergistic role in DMPS. When planning recovery actions for ensuring safety, the planner utilizes the neural policy to estimate long-term rewards, allowing it to observe beyond its short-term planning horizon. Conversely, the neural policy under training learns from the recovery plans proposed by the planner, converging to policies that are both high-performing and safe in practice. This approach guarantees safety during and after training, with bounded recovery regret that decreases exponentially with planning horizon depth. Experimental results demonstrate that DMPS converges to policies that rarely require shield interventions after training and achieve higher rewards compared to several state-of-the-art baselines.