Variance-reduced $Q$-learning is minimax optimal

TL;DR

This paper presents a variance-reduced $Q$-learning algorithm that achieves near-optimal sample complexity for estimating the optimal $Q$-function in finite MDPs, matching minimax lower bounds up to a logarithmic factor.

Contribution

It introduces a variance-reduced $Q$-learning method with provable minimax optimal sample complexity, improving upon the quartic scaling of ordinary $Q$-learning.

Findings

Achieves $ ilde{O}(rac{D}{ ext{epsilon}^2 (1- ext{gamma})^3})$ sample complexity.

Matches known minimax lower bounds up to a logarithmic factor.

Outperforms ordinary $Q$-learning with quartic scaling in discount complexity.

Abstract

We introduce and analyze a form of variance-reduced $Q$-learning. For $\gamma$-discounted MDPs with finite state space $\mathcal{X}$ and action space $\mathcal{U}$, we prove that it yields an $\epsilon$-accurate estimate of the optimal $Q$-function in the $\ell_\infty$-norm using $\mathcal{O} \left(\left(\frac{D}{ \epsilon^2 (1-\gamma)^3} \right) \; \log \left( \frac{D}{(1-\gamma)} \right) \right)$ samples, where $D = |\mathcal{X}| \times |\mathcal{U}|$. This guarantee matches known minimax lower bounds up to a logarithmic factor in the discount complexity. In contrast, our past work shows that ordinary $Q$-learning has worst-case quartic scaling in the discount complexity.