Progressive Safeguards for Safe and Model-Agnostic Reinforcement Learning

TL;DR

This paper introduces a formal, model-agnostic meta-learning framework for safe reinforcement learning that uses a safeguard modeled as a finite-state machine to ensure safety across tasks, enabling efficient transfer of safety knowledge.

Contribution

The paper presents a novel, flexible safety framework for reinforcement learning that is model-agnostic, capable of handling complex safety specifications, and transferable across tasks with minimal violations.

Findings

Agents achieve near-minimal safety violations in experiments

The framework is applicable from pixel-level control to language models

Baseline methods underperform compared to the proposed approach

Abstract

In this paper we propose a formal, model-agnostic meta-learning framework for safe reinforcement learning. Our framework is inspired by how parents safeguard their children across a progression of increasingly riskier tasks, imparting a sense of safety that is carried over from task to task. We model this as a meta-learning process where each task is synchronized with a safeguard that monitors safety and provides a reward signal to the agent. The safeguard is implemented as a finite-state machine based on a safety specification; the reward signal is formally shaped around this specification. The safety specification and its corresponding safeguard can be arbitrarily complex and non-Markovian, which adds flexibility to the training process and explainability to the learned policy. The design of the safeguard is manual but it is high-level and model-agnostic, which gives rise to an end-to-end safe learning approach with wide applicability, from pixel-level game control to language model fine-tuning. Starting from a given set of safety specifications (tasks), we train a model such that it can adapt to new specifications using only a small number of training samples. This is made possible by our method for efficiently transferring safety bias between tasks, which effectively minimizes the number of safety violations. We evaluate our framework in a Minecraft-inspired Gridworld, a VizDoom game environment, and an LLM fine-tuning application. Agents trained with our approach achieve near-minimal safety violations, while baselines are shown to underperform.