Markov Decision Processes with Value-at-Risk Criterion

TL;DR

This paper develops a novel framework for optimizing Value-at-Risk (VaR) in Markov decision processes, transforming the problem into probabilistic minimization MDPs and providing algorithms with proven convergence.

Contribution

It introduces a bilevel optimization approach for VaR in MDPs, establishing policy optimality conditions and developing convergent algorithms for both steady-state and finite-horizon scenarios.

Findings

Efficient algorithms for VaR maximization in MDPs.

Proven optimality of deterministic policies for steady-state VaR.

Numerical experiments demonstrate practical applicability.

Abstract

Value-at-risk (VaR), also known as quantile, is a crucial risk measure in finance and other fields. However, optimizing VaR metrics in Markov decision processes (MDPs) is challenging because VaR is non-additive and the traditional dynamic programming is inapplicable. This paper conducts a comprehensive study on VaR optimization in discrete-time finite MDPs. We consider VaR in two key scenarios: the VaR of steady-state rewards over an infinite horizon and the VaR of accumulated rewards over a finite horizon. By establishing the equivalence between the VaR maximization MDP and a series of probabilistic minimization MDPs, we transform the VaR maximization MDP into a constrained bilevel optimization problem. The inner-level is a policy optimization of minimizing the probability that MDP rewards fall below a target $\lambda$, while the outer-level is a single parameter optimization of $\lambda$, representing the target VaR. For the steady-state scenario, the probabilistic minimization MDP can be resolved using the expectation criterion of a standard MDP. In contrast, for the finite-horizon case, it can be addressed via an augmented-state MDP. We prove the optimality of deterministic stationary policies for steady-state VaR MDPs and deterministic history-dependent policies for finite-horizon VaR MDPs. We derive both a policy improvement rule and a necessary-sufficient condition for optimal policies. Furthermore, we develop policy iteration type algorithms to maximize the VaR in MDPs and prove their convergence. Our results are also extended to the counterpart of VaR minimization MDPs after appropriate modifications. Finally, we conduct numerical experiments to demonstrate the computational efficiency and practical applicability of our approach. Our study paves a novel way to explore the optimization of quantile-related metrics in MDPs through the duality between quantiles and probabilities.