Q-learning with UCB Exploration is Sample Efficient for Infinite-Horizon MDP

TL;DR

This paper demonstrates that a Q-learning algorithm with UCB exploration is sample efficient for infinite-horizon discounted MDPs, achieving near-optimal exploration complexity without a generative model.

Contribution

It adapts Q-learning with UCB exploration to infinite-horizon MDPs and proves a tighter sample complexity bound matching lower bounds, improving previous results.

Findings

Sample complexity bound of ilde{O}(SA/(\epsilon^2(1-\gamma)^7))

Improves upon previous bound of ilde{O}(SA/(\epsilon^4(1-\gamma)^8))

Matches lower bounds in ext{S}, ext{A}, and ext{ extbackslash epsilon} factors

Abstract

A fundamental question in reinforcement learning is whether model-free algorithms are sample efficient. Recently, Jin et al. \cite{jin2018q} proposed a Q-learning algorithm with UCB exploration policy, and proved it has nearly optimal regret bound for finite-horizon episodic MDP. In this paper, we adapt Q-learning with UCB-exploration bonus to infinite-horizon MDP with discounted rewards \emph{without} accessing a generative model. We show that the \textit{sample complexity of exploration} of our algorithm is bounded by $\tilde{O}({\frac{SA}{\epsilon^2(1-\gamma)^7}})$. This improves the previously best known result of $\tilde{O}({\frac{SA}{\epsilon^4(1-\gamma)^8}})$ in this setting achieved by delayed Q-learning \cite{strehl2006pac}, and matches the lower bound in terms of $\epsilon$ as well as $S$ and $A$ except for logarithmic factors.