Convex Approximations of Random Constrained Markov Decision Processes

TL;DR

This paper develops convex approximation methods for solving joint chance-constrained Markov decision processes with random costs and transition probabilities, providing bounds and numerical validation.

Contribution

It introduces convex upper and lower bounds for uncertain CMDPs with random costs and transitions, extending existing methods to stochastic settings.

Findings

Convex bounds effectively approximate uncertain CMDPs.

Bounds' quality validated through queueing and Garnet MDP experiments.

Proposed methods handle dependencies via Gumbel-Hougaard copula.

Abstract

Constrained Markov decision processes (CMDPs) are used as a decision-making framework to study the long-run performance of a stochastic system. It is well-known that a stationary optimal policy of a CMDP problem under discounted cost criterion can be obtained by solving a linear programming problem when running costs and transition probabilities are exactly known. In this paper, we consider a discounted cost CMDP problem where the running costs and transition probabilities are defined using random variables. Consequently, both the objective function and constraints become random. We use chance constraints to model these uncertainties and formulate the uncertain CMDP problem as a joint chance-constrained Markov decision process (JCCMDP). Under random running costs, we assume that the dependency among random constraint vectors is driven by a Gumbel-Hougaard copula. Using standard probability inequalities, we construct convex upper bound approximations of the JCCMDP problem under certain conditions on random running costs. In addition, we propose a linear programming problem whose optimal value gives a lower bound to the optimal value of the JCCMDP problem. When both running costs and transition probabilities are random, we define the latter variables as a sum of their means and random perturbations. Under mild conditions on the random perturbations and random running costs, we construct convex upper and lower bound approximations of the JCCMDP problem. We analyse the quality of the derived bounds through numerical experiments on a queueing control problem for random running costs. For the case when both running costs and transition probabilities are random, we choose randomly generated Markov decision problems called Garnets for numerical experiments.