Risk-Sensitive Markov Decision Processes with Long-Run CVaR Criterion

TL;DR

This paper develops a novel framework for optimizing long-run CVaR in Markov decision processes, introducing new formulas, optimality conditions, and an algorithm with applications in portfolio management.

Contribution

It introduces a pseudo CVaR metric, a CVaR difference formula, and a policy iteration algorithm for long-run CVaR optimization in MDPs, addressing a challenging risk metric.

Findings

Derived a CVaR difference formula for policy comparison.

Established a Bellman local optimality equation for CVaR.

Developed a convergent policy iteration algorithm for CVaR optimization.

Abstract

CVaR (Conditional Value at Risk) is a risk metric widely used in finance. However, dynamically optimizing CVaR is difficult since it is not a standard Markov decision process (MDP) and the principle of dynamic programming fails. In this paper, we study the infinite-horizon discrete-time MDP with a long-run CVaR criterion, from the view of sensitivity-based optimization. By introducing a pseudo CVaR metric, we derive a CVaR difference formula which quantifies the difference of long-run CVaR under any two policies. The optimality of deterministic policies is derived. We obtain a so-called Bellman local optimality equation for CVaR, which is a necessary and sufficient condition for local optimal policies and only necessary for global optimal policies. A CVaR derivative formula is also derived for providing more sensitivity information. Then we develop a policy iteration type algorithm to efficiently optimize CVaR, which is shown to converge to local optima in the mixed policy space. We further discuss some extensions including the mean-CVaR optimization and the maximization of CVaR. Finally, we conduct numerical experiments relating to portfolio management to demonstrate the main results. Our work may shed light on dynamically optimizing CVaR from a sensitivity viewpoint.