Efficient $Q$-Learning and Actor-Critic Methods for Robust Average Reward Reinforcement Learning

TL;DR

This paper develops non-asymptotic convergence guarantees for robust $Q$-learning and actor-critic algorithms in average reward MDPs under distributional uncertainties, enabling efficient robust policy learning.

Contribution

It introduces a novel contraction property of the robust $Q$ operator and provides sample-efficient algorithms for robust policy optimization under TV and Wasserstein uncertainties.

Findings

Optimal robust $Q$-operator is a strict contraction.

Algorithms achieve $ ilde{O}( ext{epsilon}^{-2})$ sample complexity.

Numerical simulations demonstrate effectiveness.

Abstract

We present a non-asymptotic convergence analysis of $Q$-learning and actor-critic algorithms for robust average-reward Markov Decision Processes (MDPs) under contamination, total-variation (TV) distance, and Wasserstein uncertainty sets. A key ingredient of our analysis is showing that the optimal robust $Q$ operator is a strict contraction with respect to a carefully designed semi-norm (with constant functions quotiented out). This property enables a stochastic approximation update that learns the optimal robust $Q$-function using $\tilde{\mathcal{O}}(\epsilon^{-2})$ samples. We also provide an efficient routine for robust $Q$-function estimation, which in turn facilitates robust critic estimation. Building on this, we introduce an actor-critic algorithm that learns an $\epsilon$-optimal robust policy within $\tilde{\mathcal{O}}(\epsilon^{-2})$ samples. We provide numerical simulations to evaluate the performance of our algorithms.