Risk-sensitive Actor-Critic with Static Spectral Risk Measures for Online and Offline Reinforcement Learning

TL;DR

This paper introduces a novel framework for optimizing static Spectral Risk Measures in reinforcement learning, improving risk-sensitive policies in online and offline settings with theoretical guarantees and empirical success.

Contribution

It proposes a new approach for static SRM optimization in RL, providing convergence guarantees and outperforming existing risk-sensitive methods.

Findings

Consistent outperformance of existing methods in diverse domains

Theoretical convergence guarantees in finite settings

Effective risk-sensitive policies for critical scenarios

Abstract

The development of Distributional Reinforcement Learning (DRL) has introduced a natural way to incorporate risk sensitivity into value-based and actor-critic methods by employing risk measures other than expectation in the value function. While this approach is widely adopted in many online and offline RL algorithms due to its simplicity, the naive integration of risk measures often results in suboptimal policies. This limitation can be particularly harmful in scenarios where the need for effective risk-sensitive policies is critical and worst-case outcomes carry severe consequences. To address this challenge, we propose a novel framework for optimizing static Spectral Risk Measures (SRM), a flexible family of risk measures that generalizes objectives such as CVaR and Mean-CVaR, and enables the tailoring of risk preferences. Our method is applicable to both online and offline RL algorithms. We establish theoretical guarantees by proving convergence in the finite state-action setting. Moreover, through extensive empirical evaluations, we demonstrate that our algorithms consistently outperform existing risk-sensitive methods in both online and offline environments across diverse domains.