Span-Agnostic Optimal Sample Complexity and Oracle Inequalities for Average-Reward RL

TL;DR

This paper introduces new algorithms for average-reward MDPs that achieve optimal sample complexity without prior knowledge of the span of the bias function, using innovative horizon calibration and span penalization techniques.

Contribution

The authors develop the first algorithms matching the optimal span-based complexity without prior knowledge of the span, advancing the theoretical understanding of sample efficiency in average-reward RL.

Findings

Algorithms achieve minimax optimal complexity without knowing $H$

Horizon calibration effectively tunes the effective horizon

Span penalization can outperform minimax complexity in certain settings

Abstract

We study the sample complexity of finding an $\varepsilon$-optimal policy in average-reward Markov Decision Processes (MDPs) with a generative model. The minimax optimal span-based complexity of $\widetilde{O}(SAH/\varepsilon^2)$, where $H$ is the span of the optimal bias function, has only been achievable with prior knowledge of the value of $H$. Prior-knowledge-free algorithms have been the objective of intensive research, but several natural approaches provably fail to achieve this goal. We resolve this problem, developing the first algorithms matching the optimal span-based complexity without $H$ knowledge, both when the dataset size is fixed and when the suboptimality level $\varepsilon$ is fixed. Our main technique combines the discounted reduction approach with a method for automatically tuning the effective horizon based on empirical confidence intervals or lower bounds on performance, which we term horizon calibration. We also develop an empirical span penalization approach, inspired by sample variance penalization, which satisfies an oracle inequality performance guarantee. In particular this algorithm can outperform the minimax complexity in benign settings such as when there exist near-optimal policies with span much smaller than $H$.