Algorithms for CVaR Optimization in MDPs

TL;DR

This paper develops gradient-based algorithms for optimizing risk-sensitive policies in Markov Decision Processes using the CVaR measure, with proven convergence and practical demonstration.

Contribution

It introduces new policy gradient and actor-critic algorithms for mean-CVaR optimization in MDPs, including gradient computation and convergence analysis.

Findings

Algorithms converge to locally risk-sensitive optimal policies.

Demonstrated effectiveness in an optimal stopping problem.

Provides practical methods for risk-aware decision making.

Abstract

In many sequential decision-making problems we may want to manage risk by minimizing some measure of variability in costs in addition to minimizing a standard criterion. Conditional value-at-risk (CVaR) is a relatively new risk measure that addresses some of the shortcomings of the well-known variance-related risk measures, and because of its computational efficiencies has gained popularity in finance and operations research. In this paper, we consider the mean-CVaR optimization problem in MDPs. We first derive a formula for computing the gradient of this risk-sensitive objective function. We then devise policy gradient and actor-critic algorithms that each uses a specific method to estimate this gradient and updates the policy parameters in the descent direction. We establish the convergence of our algorithms to locally risk-sensitive optimal policies. Finally, we demonstrate the usefulness of our algorithms in an optimal stopping problem.