Data-driven robust Markov decision processes on Borel spaces: performance guarantees via an axiomatic approach

TL;DR

This paper develops a data-driven approach for robust Markov decision processes on Borel spaces, providing performance guarantees and convergence analysis using an axiomatic framework for ambiguity sets based on distribution distances.

Contribution

It introduces a novel axiomatic approach to construct ambiguity sets in RMDPs, establishing convergence, probabilistic bounds, and performance guarantees with finite samples.

Findings

Robust optimal value functions converge to true values as sample size increases.

Finite sample robust value bounds serve as high probability upper bounds on out-of-sample performance.

Several well-known distances satisfy the concentration inequalities needed for performance guarantees.

Abstract

We consider Markov decision processes (MDPs) with unknown disturbance distribution and address this problem using the robust Markov decision process (RMDP) approach. We construct the empirical distribution of the unknown disturbance distribution and characterize our ambiguity set of distributions as the sublevel set of a nonnegative distance function from the empirical distribution. By connecting the weak convergence of distributions to convergence with respect to the distance function, we prove that the robust optimal value function and the out-of-sample value function converge to the true optimal value function with increasing sample-sizes. We establish that, for finite sample-sizes, the robust optimal value function serves as a high probability upper bound on the out-of-sample value function. We also obtain probabilistic convergence rates, sample complexity bounds, and out-of-distribution performance bounds. The finite sample performance guarantees rely on the distance function satisfying a certain concentration type inequality. Several well-studied distances in the literature meet the requirements imposed on the distance function. We also analyze the data-driven properties of empirical MDPs and demonstrate that, unlike our data-driven RMDPs, empirical MDPs fail to satisfy some of the finite sample performance guarantees.