Safety through Permissibility: Shield Construction for Fast and Safe Reinforcement Learning

TL;DR

This paper introduces a permissibility-based framework for reinforcement learning that ensures safety while improving training efficiency, addressing limitations of existing shielding methods.

Contribution

It extends permissibility to incorporate safety constraints, enabling fast, safe RL without extensive human effort or pre-computation.

Findings

Effective safety enforcement demonstrated in three RL applications

Improved training efficiency over traditional shielding methods

Framework reduces the need for extensive domain modeling

Abstract

Designing Reinforcement Learning (RL) solutions for real-life problems remains a significant challenge. A major area of concern is safety. "Shielding" is a popular technique to enforce safety in RL by turning user-defined safety specifications into safe agent behavior. However, these methods either suffer from extreme learning delays, demand extensive human effort in designing models and safe domains in the problem, or require pre-computation. In this paper, we propose a new permissibility-based framework to deal with safety and shield construction. Permissibility was originally designed for eliminating (non-permissible) actions that will not lead to an optimal solution to improve RL training efficiency. This paper shows that safety can be naturally incorporated into this framework, i.e. extending permissibility to include safety, and thereby we can achieve both safety and improved efficiency. Experimental evaluation using three standard RL applications shows the effectiveness of the approach.