Finite-Sample Analysis of Policy Evaluation for Robust Average Reward Reinforcement Learning

TL;DR

This paper provides the first finite-sample analysis for policy evaluation in robust average-reward MDPs, establishing sample complexity bounds and introducing a novel MLMC-based estimation method with finite expected samples.

Contribution

It introduces a finite-sample analysis framework for robust average-reward policy evaluation and develops a new MLMC-based estimator with controlled bias and finite expected samples.

Findings

Achieves order-optimal sample complexity of ~O(ε^{-2})

Introduces a truncation mechanism ensuring finite expected samples

Establishes the contraction property of the robust Bellman operator

Abstract

We present the first finite-sample analysis of policy evaluation in robust average-reward Markov Decision Processes (MDPs). Prior work in this setting have established only asymptotic convergence guarantees, leaving open the question of sample complexity. In this work, we address this gap by showing that the robust Bellman operator is a contraction under a carefully constructed semi-norm, and developing a stochastic approximation framework with controlled bias. Our approach builds upon Multi-Level Monte Carlo (MLMC) techniques to estimate the robust Bellman operator efficiently. To overcome the infinite expected sample complexity inherent in standard MLMC, we introduce a truncation mechanism based on a geometric distribution, ensuring a finite expected sample complexity while maintaining a small bias that decays exponentially with the truncation level. Our method achieves the order-optimal sample complexity of $\tilde{\mathcal{O}}(\epsilon^{-2})$ for robust policy evaluation and robust average reward estimation, marking a significant advancement in robust reinforcement learning theory.