Sample Complexity of Distributionally Robust Average-Reward Reinforcement Learning

TL;DR

This paper introduces two algorithms for distributionally robust average-reward reinforcement learning, providing near-optimal sample complexity guarantees and the first finite-sample convergence analysis for this setting.

Contribution

The paper proposes two novel algorithms for DR average-reward RL with proven near-optimal sample complexity and convergence guarantees under certain conditions.

Findings

Both algorithms achieve a sample complexity of  O(|S||A| t_{mix}^2 \u03b5^{-2})

First finite-sample convergence guarantee for DR average-reward RL

Algorithms validated through numerical experiments

Abstract

Motivated by practical applications where stable long-term performance is critical-such as robotics, operations research, and healthcare-we study the problem of distributionally robust (DR) average-reward reinforcement learning. We propose two algorithms that achieve near-optimal sample complexity. The first reduces the problem to a DR discounted Markov decision process (MDP), while the second, Anchored DR Average-Reward MDP, introduces an anchoring state to stabilize the controlled transition kernels within the uncertainty set. Assuming the nominal MDP is uniformly ergodic, we prove that both algorithms attain a sample complexity of $\widetilde{O}\left(|\mathbf{S}||\mathbf{A}| t_{\mathrm{mix}}^2\varepsilon^{-2}\right)$ for estimating the optimal policy as well as the robust average reward under KL and $f_k$-divergence-based uncertainty sets, provided the uncertainty radius is sufficiently small. Here, $\varepsilon$ is the target accuracy, $|\mathbf{S}|$ and $|\mathbf{A}|$ denote the sizes of the state and action spaces, and $t_{\mathrm{mix}}$ is the mixing time of the nominal MDP. This represents the first finite-sample convergence guarantee for DR average-reward reinforcement learning. We further validate the convergence rates of our algorithms through numerical experiments.