A Finite Sample Complexity Bound for Distributionally Robust Q-learning

TL;DR

This paper establishes the first finite sample complexity bounds for distributionally robust Q-learning in reinforcement learning, demonstrating how to efficiently learn robust policies under environment uncertainty with theoretical guarantees.

Contribution

It extends the robust Q-learning framework with improved estimator design and provides the first known sample complexity bounds for model-free robust reinforcement learning.

Findings

Sample complexity bound depends on state-action space, discount factor, and uncertainty parameters.

Algorithm achieves epsilon-accuracy with high probability within the derived sample complexity.

Simulation results support the theoretical analysis and effectiveness of the proposed method.

Abstract

We consider a reinforcement learning setting in which the deployment environment is different from the training environment. Applying a robust Markov decision processes formulation, we extend the distributionally robust $Q$-learning framework studied in Liu et al. [2022]. Further, we improve the design and analysis of their multi-level Monte Carlo estimator. Assuming access to a simulator, we prove that the worst-case expected sample complexity of our algorithm to learn the optimal robust $Q$-function within an $\epsilon$ error in the sup norm is upper bounded by $\tilde O(|S||A|(1-\gamma)^{-5}\epsilon^{-2}p_{\wedge}^{-6}\delta^{-4})$, where $\gamma$ is the discount rate, $p_{\wedge}$ is the non-zero minimal support probability of the transition kernels and $\delta$ is the uncertainty size. This is the first sample complexity result for the model-free robust RL problem. Simulation studies further validate our theoretical results.