Provably Adaptive Average Reward Reinforcement Learning for Metric Spaces

TL;DR

This paper introduces ZoRL, an adaptive reinforcement learning algorithm for Lipschitz MDPs that achieves better regret bounds by zooming into promising regions, outperforming fixed discretization methods.

Contribution

The paper develops ZoRL, an adaptive algorithm that improves regret bounds in average-reward RL for Lipschitz MDPs by dynamically discretizing the state-action space.

Findings

ZoRL achieves regret bounds of $ ilde{O}(T^{1 - d_{eff}^{-1}})$, improving over fixed discretization methods.

ZoRL outperforms state-of-the-art algorithms in experiments, demonstrating the benefits of adaptivity.

The algorithm effectively captures problem-specific structure through the zooming dimension, leading to smaller regret in benign MDPs.

Abstract

We study infinite-horizon average-reward reinforcement learning (RL) for Lipschitz MDPs, a broad class that subsumes several important classes such as linear and RKHS MDPs, function approximation frameworks, and develop an adaptive algorithm $\text{ZoRL}$ with regret bounded as $\mathcal{O}\big(T^{1 - d_{\text{eff.}}^{-1}}\big)$, where $d_{\text{eff.}}= 2d_\mathcal{S} + d_z + 3$, $d_\mathcal{S}$ is the dimension of the state space and $d_z$ is the zooming dimension. In contrast, algorithms with fixed discretization yield $d_{\text{eff.}} = 2(d_\mathcal{S} + d_\mathcal{A}) + 2$, $d_\mathcal{A}$ being the dimension of action space. $\text{ZoRL}$ achieves this by discretizing the state-action space adaptively and zooming into ''promising regions'' of the state-action space. $d_z$, a problem-dependent quantity bounded by the state-action space's dimension, allows us to conclude that if an MDP is benign, then the regret of $\text{ZoRL}$ will be small. The zooming dimension and $\text{ZoRL}$ are truly adaptive, i.e., the current work shows how to capture adaptivity gains for infinite-horizon average-reward RL. $\text{ZoRL}$ outperforms other state-of-the-art algorithms in experiments, thereby demonstrating the gains arising due to adaptivity.