Predictive Safety Shield for Dyna-Q Reinforcement Learning

TL;DR

This paper introduces a predictive safety shield for model-based reinforcement learning that enhances safety and performance by using safe environment predictions, demonstrated effectively in gridworld environments.

Contribution

It proposes a novel safety shield that updates the Q-function based on safe predictions, improving safety and performance without extra training.

Findings

Short prediction horizons can identify optimal paths.

The approach is robust to distribution shifts.

Maintains safety guarantees while improving performance.

Abstract

Obtaining safety guarantees for reinforcement learning is a major challenge to achieve applicability for real-world tasks. Safety shields extend standard reinforcement learning and achieve hard safety guarantees. However, existing safety shields commonly use random sampling of safe actions or a fixed fallback controller, therefore disregarding future performance implications of different safe actions. In this work, we propose a predictive safety shield for model-based reinforcement learning agents in discrete space. Our safety shield updates the Q-function locally based on safe predictions, which originate from a safe simulation of the environment model. This shielding approach improves performance while maintaining hard safety guarantees. Our experiments on gridworld environments demonstrate that even short prediction horizons can be sufficient to identify the optimal path. We observe that our approach is robust to distribution shifts, e.g., between simulation and reality, without requiring additional training.