Almost Optimal Model-Free Reinforcement Learning via Reference-Advantage Decomposition

TL;DR

This paper introduces a model-free reinforcement learning algorithm, UCB-Advantage, that achieves near-optimal regret bounds in finite-horizon MDPs, matching the performance of model-based methods and lower bounds.

Contribution

The paper presents UCB-Advantage, a novel model-free RL algorithm with improved regret bounds and applicability to concurrent learning, surpassing previous methods.

Findings

Achieves $ ilde{O}( oot{2}H^2SAT)$ regret bound

Matches the best known model-based algorithms and lower bounds

Has low local switching cost and supports concurrent RL

Abstract

We study the reinforcement learning problem in the setting of finite-horizon episodic Markov Decision Processes (MDPs) with $S$ states, $A$ actions, and episode length $H$. We propose a model-free algorithm UCB-Advantage and prove that it achieves $\tilde{O}(\sqrt{H^2SAT})$ regret where $T = KH$ and $K$ is the number of episodes to play. Our regret bound improves upon the results of [Jin et al., 2018] and matches the best known model-based algorithms as well as the information theoretic lower bound up to logarithmic factors. We also show that UCB-Advantage achieves low local switching cost and applies to concurrent reinforcement learning, improving upon the recent results of [Bai et al., 2019].