A Statistical Analysis of Polyak-Ruppert Averaged Q-learning

TL;DR

This paper provides a detailed statistical analysis of Polyak-Ruppert averaged Q-learning, establishing a functional central limit theorem, online inference methods, and optimal error bounds in tabular Markov decision processes.

Contribution

It introduces a functional CLT for averaged Q-learning, shows its asymptotic efficiency as an RAL estimator, and derives nonasymptotic error bounds, extending to entropy-regularized Q-learning.

Findings

Functional CLT for averaged Q-learning process

Online inference method based on the CLT

Instance-dependent lower bounds for error

Abstract

We study Q-learning with Polyak-Ruppert averaging in a discounted Markov decision process in synchronous and tabular settings. Under a Lipschitz condition, we establish a functional central limit theorem for the averaged iteration $\bar{\boldsymbol{Q}}_T$ and show that its standardized partial-sum process converges weakly to a rescaled Brownian motion. The functional central limit theorem implies a fully online inference method for reinforcement learning. Furthermore, we show that $\bar{\boldsymbol{Q}}_T$ is the regular asymptotically linear (RAL) estimator for the optimal Q-value function $\boldsymbol{Q}^*$ that has the most efficient influence function. We present a nonasymptotic analysis for the $\ell_{\infty}$ error, $\mathbb{E}\|\bar{\boldsymbol{Q}}_T-\boldsymbol{Q}^*\|_{\infty}$, showing that it matches the instance-dependent lower bound for polynomial step sizes. Similar results are provided for entropy-regularized Q-learning without the Lipschitz condition.