Efficient Risk-sensitive Planning via Entropic Risk Measures

TL;DR

This paper introduces a novel method for efficiently computing optimal policies in risk-sensitive planning using entropic risk measures, providing tight approximations for tail-focused metrics in MDPs with strong empirical performance.

Contribution

It presents a new structural analysis and smoothness properties that enable effective computation of the entire optimal policy set for entropic risk measures across parameters.

Findings

Effective approximation of tail-focused risk metrics.

Strong empirical performance across decision scenarios.

Efficient computation of optimal policies for entropic risks.

Abstract

Risk-sensitive planning aims to identify policies maximizing some tail-focused metrics in Markov Decision Processes (MDPs). Such an optimization task can be very costly for the most widely used and interpretable metrics such as threshold probabilities or (Conditional) Values at Risk. Indeed, previous work showed that only Entropic Risk Measures (EntRM) can be efficiently optimized through dynamic programming, leaving a hard-to-interpret parameter to choose. We show that the computation of the full set of optimal policies for EntRM across parameter values leads to tight approximations for the metrics of interest. We prove that this optimality front can be computed effectively thanks to a novel structural analysis and smoothness properties of entropic risks. Empirical results demonstrate that our approach achieves strong performance in a variety of decision-making scenarios.