Robust $Q$-learning Algorithm for Markov Decision Processes under Wasserstein Uncertainty

TL;DR

This paper introduces a new $Q$-learning algorithm designed for distributionally robust Markov decision processes with Wasserstein ambiguity sets, demonstrating convergence and practical benefits through real data examples.

Contribution

The paper develops a novel $Q$-learning method for Wasserstein distributional robustness in MDPs, with proven convergence and demonstrated practical advantages.

Findings

Algorithm converges reliably in tested scenarios.

Distributional robustness improves decision quality under model misspecification.

Real data examples show the algorithm's tractability and robustness benefits.

Abstract

We present a novel $Q$-learning algorithm tailored to solve distributionally robust Markov decision problems where the corresponding ambiguity set of transition probabilities for the underlying Markov decision process is a Wasserstein ball around a (possibly estimated) reference measure. We prove convergence of the presented algorithm and provide several examples also using real data to illustrate both the tractability of our algorithm as well as the benefits of considering distributional robustness when solving stochastic optimal control problems, in particular when the estimated distributions turn out to be misspecified in practice.