A Survey of Constraint Formulations in Safe Reinforcement Learning

TL;DR

This paper provides a comprehensive review of various constraint formulations in safe reinforcement learning, analyzing their interrelations, algorithms, and theoretical foundations to advance understanding and future research directions.

Contribution

It systematically categorizes constraint formulations in safe RL, explores their interrelations, and discusses algorithms and theoretical insights, filling a significant knowledge gap.

Findings

Identifies diverse constraint representations in safe RL.

Elucidates mathematical relations among common formulations.

Discusses current challenges and future research directions.

Abstract

Safety is critical when applying reinforcement learning (RL) to real-world problems. As a result, safe RL has emerged as a fundamental and powerful paradigm for optimizing an agent's policy while incorporating notions of safety. A prevalent safe RL approach is based on a constrained criterion, which seeks to maximize the expected cumulative reward subject to specific safety constraints. Despite recent effort to enhance safety in RL, a systematic understanding of the field remains difficult. This challenge stems from the diversity of constraint representations and little exploration of their interrelations. To bridge this knowledge gap, we present a comprehensive review of representative constraint formulations, along with a curated selection of algorithms designed specifically for each formulation. In addition, we elucidate the theoretical underpinnings that reveal the mathematical mutual relations among common problem formulations. We conclude with a discussion of the current state and future directions of safe reinforcement learning research.