Risk-Averse Constrained Reinforcement Learning with Optimized Certainty Equivalents

TL;DR

This paper introduces a risk-aware constrained reinforcement learning framework using optimized certainty equivalents, addressing tail risks and catastrophic events in high-stakes scenarios, with proven convergence and practical algorithms.

Contribution

It develops a novel risk-aware constrained RL method with a strong duality framework and practical algorithms compatible with standard RL solvers.

Findings

Ensures risk-aware properties through numerical experiments

Provides convergence guarantees for the proposed algorithm

Demonstrates applicability with standard RL algorithms like PPO

Abstract

Constrained optimization provides a common framework for dealing with conflicting objectives in reinforcement learning (RL). In most of these settings, the objectives (and constraints) are expressed though the expected accumulated reward. However, this formulation neglects risky or even possibly catastrophic events at the tails of the reward distribution, and is often insufficient for high-stakes applications in which the risk involved in outliers is critical. In this work, we propose a framework for risk-aware constrained RL, which exhibits per-stage robustness properties jointly in reward values and time using optimized certainty equivalents (OCEs). Our framework ensures an exact equivalent to the original constrained problem within a parameterized strong Lagrangian duality framework under appropriate constraint qualifications, and yields a simple algorithmic recipe which can be wrapped around standard RL solvers, such as PPO. Lastly, we establish the convergence of the proposed algorithm under common assumptions, and verify the risk-aware properties of our approach through several numerical experiments.