Extreme Value Policy Optimization for Safe Reinforcement Learning

TL;DR

This paper introduces EVO, a reinforcement learning algorithm that uses Extreme Value Theory to better model and mitigate rare, high-impact constraint violations, improving safety in real-world applications.

Contribution

EVO is the first RL method to explicitly incorporate extreme value modeling for safety constraints, reducing violations and providing theoretical guarantees.

Findings

EVO reduces constraint violations more effectively than baseline methods.

EVO maintains competitive policy performance.

EVO exhibits lower variance than quantile regression approaches.

Abstract

Ensuring safety is a critical challenge in applying Reinforcement Learning (RL) to real-world scenarios. Constrained Reinforcement Learning (CRL) addresses this by maximizing returns under predefined constraints, typically formulated as the expected cumulative cost. However, expectation-based constraints overlook rare but high-impact extreme value events in the tail distribution, such as black swan incidents, which can lead to severe constraint violations. To address this issue, we propose the Extreme Value policy Optimization (EVO) algorithm, leveraging Extreme Value Theory (EVT) to model and exploit extreme reward and cost samples, reducing constraint violations. EVO introduces an extreme quantile optimization objective to explicitly capture extreme samples in the cost tail distribution. Additionally, we propose an extreme prioritization mechanism during replay, amplifying the learning signal from rare but high-impact extreme samples. Theoretically, we establish upper bounds on expected constraint violations during policy updates, guaranteeing strict constraint satisfaction at a zero-violation quantile level. Further, we demonstrate that EVO achieves a lower probability of constraint violations than expectation-based methods and exhibits lower variance than quantile regression methods. Extensive experiments show that EVO significantly reduces constraint violations during training while maintaining competitive policy performance compared to baselines.