Model-free Reinforcement Learning in Infinite-horizon Average-reward Markov Decision Processes

TL;DR

This paper introduces two novel model-free reinforcement learning algorithms for infinite-horizon average-reward MDPs, achieving improved regret bounds under different assumptions, advancing the efficiency and applicability of RL in large-scale problems.

Contribution

It presents the first model-free algorithm for general MDPs with weakly communicating assumptions and enhances regret bounds for ergodic MDPs using adaptive bandit techniques.

Findings

First algorithm achieves $ ext{O}(T^{2/3})$ regret for weakly communicating MDPs.

Second algorithm improves regret to $ ext{O}( oot 2 ext{T})$ for ergodic MDPs.

Significant improvement over previous $ ext{O}(T^{3/4})$ regret results.

Abstract

Model-free reinforcement learning is known to be memory and computation efficient and more amendable to large scale problems. In this paper, two model-free algorithms are introduced for learning infinite-horizon average-reward Markov Decision Processes (MDPs). The first algorithm reduces the problem to the discounted-reward version and achieves $\mathcal{O}(T^{2/3})$ regret after $T$ steps, under the minimal assumption of weakly communicating MDPs. To our knowledge, this is the first model-free algorithm for general MDPs in this setting. The second algorithm makes use of recent advances in adaptive algorithms for adversarial multi-armed bandits and improves the regret to $\mathcal{O}(\sqrt{T})$, albeit with a stronger ergodic assumption. This result significantly improves over the $\mathcal{O}(T^{3/4})$ regret achieved by the only existing model-free algorithm by Abbasi-Yadkori et al. (2019a) for ergodic MDPs in the infinite-horizon average-reward setting.