Variance-Reduced Cascade Q-learning: Algorithms and Sample Complexity

TL;DR

This paper introduces Variance-Reduced Cascade Q-learning (VRCQ), a novel algorithm that achieves minimax optimal sample complexity for estimating optimal Q-functions in discounted MDPs, with strong theoretical guarantees and practical efficiency.

Contribution

The paper proposes VRCQ, combining variance reduction techniques to improve sample complexity and guarantee optimality in model-free Q-learning for discounted MDPs.

Findings

VRCQ achieves minimax optimal sample complexity.

VRCQ outperforms existing algorithms in $oldsymbol{ ext{l}_ extbf{ extit{infty}}}$-norm guarantees.

Numerical experiments validate theoretical results.

Abstract

We study the problem of estimating the optimal Q-function of $\gamma$-discounted Markov decision processes (MDPs) under the synchronous setting, where independent samples for all state-action pairs are drawn from a generative model at each iteration. We introduce and analyze a novel model-free algorithm called Variance-Reduced Cascade Q-learning (VRCQ). VRCQ comprises two key building blocks: (i) the established direct variance reduction technique and (ii) our proposed variance reduction scheme, Cascade Q-learning. By leveraging these techniques, VRCQ provides superior guarantees in the $\ell_\infty$-norm compared with the existing model-free stochastic approximation-type algorithms. Specifically, we demonstrate that VRCQ is minimax optimal. Additionally, when the action set is a singleton (so that the Q-learning problem reduces to policy evaluation), it achieves non-asymptotic instance optimality while requiring the minimum number of samples theoretically possible. Our theoretical results and their practical implications are supported by numerical experiments.