Optimal Sample Complexity for Average Reward Markov Decision Processes

TL;DR

This paper presents a new algorithm for policy learning in average reward Markov decision processes that achieves near-optimal sample complexity, closing the gap between existing upper and lower bounds.

Contribution

We develop the first estimator for average reward MDPs that matches the theoretical lower bound on sample complexity, improving upon previous methods.

Findings

Our algorithm attains the optimal sample complexity of rac{1}{2}(|S||A|t_{mix}\u00b2 \u2206^{-2})

Numerical experiments confirm the theoretical efficiency of the proposed method.

The approach bridges the gap between upper and lower bounds in sample complexity for average reward MDPs.

Abstract

We resolve the open question regarding the sample complexity of policy learning for maximizing the long-run average reward associated with a uniformly ergodic Markov decision process (MDP), assuming a generative model. In this context, the existing literature provides a sample complexity upper bound of $\widetilde O(|S||A|t_{\text{mix}}^2 \epsilon^{-2})$ and a lower bound of $\Omega(|S||A|t_{\text{mix}} \epsilon^{-2})$. In these expressions, $|S|$ and $|A|$ denote the cardinalities of the state and action spaces respectively, $t_{\text{mix}}$ serves as a uniform upper limit for the total variation mixing times, and $\epsilon$ signifies the error tolerance. Therefore, a notable gap of $t_{\text{mix}}$ still remains to be bridged. Our primary contribution is the development of an estimator for the optimal policy of average reward MDPs with a sample complexity of $\widetilde O(|S||A|t_{\text{mix}}\epsilon^{-2})$. This marks the first algorithm and analysis to reach the literature's lower bound. Our new algorithm draws inspiration from ideas in Li et al. (2020), Jin and Sidford (2021), and Wang et al. (2023). Additionally, we conduct numerical experiments to validate our theoretical findings.