Robust Markov decision processes under parametric transition distributions

TL;DR

This paper develops algorithms for robust Markov decision processes with parametric transition distributions, enabling efficient robust value iteration by avoiding extensive pre-computation and discretisation.

Contribution

It introduces a novel projection-based bisection search algorithm that eliminates the need for discretisation and pre-computation in robust Bellman updates.

Findings

Projection-based algorithm is significantly faster than other methods.

Algorithms perform well on multi-period newsvendor problems.

Parametric approach outperforms non-parametric methods in speed.

Abstract

This paper considers robust Markov decision processes under parametric transition distributions. We assume that the true transition distribution is uniquely specified by some parametric distribution, and explicitly enforce that the worst-case distribution from the model is uniquely specified by a distribution in the same parametric family. After formulating the parametric robust model, we focus on developing algorithms for carrying out the robust Bellman updates required to complete robust value iteration. We first formulate the update as a linear program by discretising the ambiguity set. Since this model scales poorly with problem size and requires large amounts of pre-computation, we develop two additional algorithms for solving the robust Bellman update. Firstly, we present a cutting surface algorithm for solving this linear program in a shorter time. This algorithm requires the same pre-computation, but only ever solves the linear program over small subsets of the ambiguity set. Secondly, we present a novel projection-based bisection search algorithm that completely eliminates the need for discretisation and does not require any pre-computation. We test our algorithms extensively on a dynamic multi-period newsvendor problem under binomial and Poisson demands. In addition, we compare our methods with the non-parametric phi-divergence based methods from the literature. We show that our projection-based algorithm completes robust value iteration significantly faster than our other two parametric algorithms, and also faster than its non-parametric equivalent.