Provably Sample-Efficient Robust Reinforcement Learning with Average Reward

TL;DR

This paper introduces RHI, a new algorithm for robust average-reward reinforcement learning that achieves near-optimal sample complexity with weaker assumptions and no prior knowledge, advancing theoretical understanding of data-efficient robust RL.

Contribution

The paper presents RHI, a robust RL algorithm with the weakest assumptions and optimal sample complexity, filling a key gap in finite-sample analysis for robust average-reward RL.

Findings

RHI achieves the tightest known sample complexity bound.

It requires only the communicating condition, not ergodicity.

No prior knowledge of the MDP is needed for RHI.

Abstract

Robust reinforcement learning (RL) under the average-reward criterion is essential for long-term decision-making, particularly when the environment may differ from its specification. However, a significant gap exists in understanding the finite-sample complexity of these methods, as most existing work provides only asymptotic guarantees. This limitation hinders their principled understanding and practical deployment, especially in data-limited scenarios. We close this gap by proposing \textbf{Robust Halpern Iteration (RHI)}, a new algorithm designed for robust Markov Decision Processes (MDPs) with transition uncertainty characterized by $\ell_p$-norm and contamination models. Our approach offers three key advantages over previous methods: (1). Weaker Structural Assumptions: RHI only requires the underlying robust MDP to be communicating, a less restrictive condition than the commonly assumed ergodicity or irreducibility; (2). No Prior Knowledge: Our algorithm operates without requiring any prior knowledge of the robust MDP; (3). State-of-the-Art Sample Complexity: To learn an $\epsilon$-optimal robust policy, RHI achieves a sample complexity of $\tilde{\mathcal O}\left(\frac{SA\mathcal H^{2}}{\epsilon^{2}}\right)$, where $S$ and $A$ denote the numbers of states and actions, and $\mathcal H$ is the robust optimal bias span. This result represents the tightest known bound. Our work hence provides essential theoretical understanding of sample efficiency of robust average reward RL.