Achieving Tractable Minimax Optimal Regret in Average Reward MDPs

TL;DR

This paper introduces a computationally efficient algorithm for average-reward MDPs that achieves minimax optimal regret bounds without prior knowledge of the bias span, improving upon previous methods.

Contribution

The paper presents the first tractable algorithm with minimax optimal regret for average-reward MDPs, utilizing a novel subroutine PMEVI that enhances existing algorithms.

Findings

Achieves regret of (\u007f( ext{sp}(h^*) S A T)) with high probability.

Does not require prior knowledge of the bias span (h^*).

Introduces PMEVI, a new subroutine for bias-constrained policy computation.

Abstract

In recent years, significant attention has been directed towards learning average-reward Markov Decision Processes (MDPs). However, existing algorithms either suffer from sub-optimal regret guarantees or computational inefficiencies. In this paper, we present the first tractable algorithm with minimax optimal regret of $\widetilde{\mathrm{O}}(\sqrt{\mathrm{sp}(h^*) S A T})$, where $\mathrm{sp}(h^*)$ is the span of the optimal bias function $h^*$, $S \times A$ is the size of the state-action space and $T$ the number of learning steps. Remarkably, our algorithm does not require prior information on $\mathrm{sp}(h^*)$. Our algorithm relies on a novel subroutine, Projected Mitigated Extended Value Iteration (PMEVI), to compute bias-constrained optimal policies efficiently. This subroutine can be applied to various previous algorithms to improve regret bounds.