Stochastic dominance-constrained Markov decision processes

TL;DR

This paper introduces a linear programming approach to solve risk-constrained Markov decision processes using stochastic dominance constraints, applicable to both average and discounted reward settings, with a portfolio optimization example.

Contribution

It develops a novel linear programming formulation for stochastic dominance-constrained MDPs, including dual dynamic programming equations with a new pricing term, extending to various stochastic orders.

Findings

Linear constraints on occupation measures for risk constraints.

Optimal policies derived from linear programs incorporating dominance constraints.

Application demonstrated in portfolio optimization.

Abstract

We are interested in risk constraints for infinite horizon discrete time Markov decision processes (MDPs). Starting with average reward MDPs, we show that increasing concave stochastic dominance constraints on the empirical distribution of reward lead to linear constraints on occupation measures. The optimal policy for the resulting class of dominance-constrained MDPs is obtained by solving a linear program. We compute the dual of this linear program to obtain average dynamic programming optimality equations that reflect the dominance constraint. In particular, a new pricing term appears in the optimality equations corresponding to the dominance constraint. We show that many types of stochastic orders can be used in place of the increasing concave stochastic order. We also carry out a parallel development for discounted reward MDPs with stochastic dominance constraints. The paper concludes with a portfolio optimization example.