On Linear Stochastic Approximation: Fine-grained Polyak-Ruppert and Non-Asymptotic Concentration

TL;DR

This paper provides a detailed analysis of linear stochastic approximation with Polyak-Ruppert averaging, establishing precise asymptotic distributions and non-asymptotic concentration bounds, with applications to reinforcement learning and optimization.

Contribution

It offers the first comprehensive CLT and concentration inequalities for linear stochastic approximation under general spectral conditions, including non-Hurwitz matrices.

Findings

CLT characterizes asymptotic covariance with correction term

Non-asymptotic bounds match CLT covariance up to constants

Achieves O(1/T) mean-squared error rate for non-Hurwitz matrices

Abstract

We undertake a precise study of the asymptotic and non-asymptotic properties of stochastic approximation procedures with Polyak-Ruppert averaging for solving a linear system $\bar{A} \theta = \bar{b}$. When the matrix $\bar{A}$ is Hurwitz, we prove a central limit theorem (CLT) for the averaged iterates with fixed step size and number of iterations going to infinity. The CLT characterizes the exact asymptotic covariance matrix, which is the sum of the classical Polyak-Ruppert covariance and a correction term that scales with the step size. Under assumptions on the tail of the noise distribution, we prove a non-asymptotic concentration inequality whose main term matches the covariance in CLT in any direction, up to universal constants. When the matrix $\bar{A}$ is not Hurwitz but only has non-negative real parts in its eigenvalues, we prove that the averaged LSA procedure actually achieves an $O(1/T)$ rate in mean-squared error. Our results provide a more refined understanding of linear stochastic approximation in both the asymptotic and non-asymptotic settings. We also show various applications of the main results, including the study of momentum-based stochastic gradient methods as well as temporal difference algorithms in reinforcement learning.