Dynamically Augmented CVaR for MDPs

TL;DR

This paper introduces the Dynamically augmented CVaR (DCVaR), a time-consistent risk measure for MDPs, along with an algorithm to optimize it, improving the calculation of risk-sensitive policies.

Contribution

It defines DCVaR as a dynamic, time-consistent version of static CVaR and develops an algorithm to optimize policies under this new risk measure.

Findings

DCVaR is a lower bound of static CVaR.

The algorithm optimizes policies for total discounted costs under DCVaR.

The correctness is proved via a mass transfer problem.

Abstract

This paper studies optimization of Conditional Value-at-Risk (CVaR) for Markov Decision Processes (MDPs) with finite state and action sets. It introduces the Dynamically augmented CVaR (DCVaR) risk measure and provides an algorithm for its optimization. This paper investigates a specially defined Robust MDP (RMDP), in which the state space is augmented with the tail risk level. This RMDP, which we call the Dynamically augmented RMDP (DRMDP), was introduced to the literature for calculations of optimal CVaR values by value iteration more than ten years ago, but, as was understood later, these value iterations compute lower bounds of minimal static CVaRs. DCVaR is defined as a time consistent version of the static CVaR, and it is a lower bound of the static CVaR. It also can be considered as a dynamic version of the nested CVaR. This paper provides an algorithm constructing a policy optimizing DCVaR of total discounted costs. The correctness of this algorithm is proved by studying a special mass transfer problem. The results on RMDPs needed for this paper are provided in the appendix.