Bounding the Difference between the Values of Robust and Non-Robust Markov Decision Problems

TL;DR

This paper establishes a dimension-free upper bound on the difference in value functions between distributionally robust and non-robust Markov decision processes, based on Wasserstein-ball ambiguity sets.

Contribution

It provides a novel, linear, and dimension-free upper bound for the value difference in robust MDPs with Wasserstein ambiguity sets.

Findings

Upper bound is dimension-free

Difference depends linearly on Wasserstein radius

Applicable to distributionally robust MDPs

Abstract

In this note we provide an upper bound for the difference between the value function of a distributionally robust Markov decision problem and the value function of a non-robust Markov decision problem, where the ambiguity set of probability kernels of the distributionally robust Markov decision process is described by a Wasserstein-ball around some reference kernel whereas the non-robust Markov decision process behaves according to a fixed probability kernel contained in the ambiguity set. Our derived upper bound for the difference between the value functions is dimension-free and depends linearly on the radius of the Wasserstein-ball.