Extreme Risk Mitigation in Reinforcement Learning using Extreme Value Theory

TL;DR

This paper introduces a novel risk-sensitive reinforcement learning approach that employs extreme value theory to better model rare, catastrophic risk events, improving resilience and outperforming existing methods on benchmark tasks.

Contribution

It proposes a new EVT-based parameterization of the value distribution to enhance modeling of rare risky events in RL, with theoretical and empirical validation.

Findings

Outperforms existing risk-averse RL algorithms on benchmarks

Effectively models rare catastrophic events

Provides theoretical advantages of EVT-based distributions

Abstract

Risk-sensitive reinforcement learning (RL) has garnered significant attention in recent years due to the growing interest in deploying RL agents in real-world scenarios. A critical aspect of risk awareness involves modeling highly rare risk events (rewards) that could potentially lead to catastrophic outcomes. These infrequent occurrences present a formidable challenge for data-driven methods aiming to capture such risky events accurately. While risk-aware RL techniques do exist, their level of risk aversion heavily relies on the precision of the state-action value function estimation when modeling these rare occurrences. Our work proposes to enhance the resilience of RL agents when faced with very rare and risky events by focusing on refining the predictions of the extreme values predicted by the state-action value function distribution. To achieve this, we formulate the extreme values of the state-action value function distribution as parameterized distributions, drawing inspiration from the principles of extreme value theory (EVT). This approach effectively addresses the issue of infrequent occurrence by leveraging EVT-based parameterization. Importantly, we theoretically demonstrate the advantages of employing these parameterized distributions in contrast to other risk-averse algorithms. Our evaluations show that the proposed method outperforms other risk averse RL algorithms on a diverse range of benchmark tasks, each encompassing distinct risk scenarios.