Statistical inference for Linear Stochastic Approximation with Markovian Noise

TL;DR

This paper provides non-asymptotic bounds and bootstrap validity results for the statistical inference of Linear Stochastic Approximation algorithms with Markovian noise, including convergence rates and confidence interval guarantees.

Contribution

It introduces the first non-asymptotic guarantees for bootstrap-based confidence intervals in Markovian stochastic approximation.

Findings

Established $ ext{O}(n^{-1/4})$ convergence rates to Gaussian limit.

Proved the validity of multiplier block bootstrap for confidence intervals.

Recovered classical $ ext{O}(n^{-1/8})$ rate for asymptotic variance estimation.

Abstract

In this paper we derive non-asymptotic Berry-Esseen bounds for Polyak-Ruppert averaged iterates of the Linear Stochastic Approximation (LSA) algorithm driven by the Markovian noise. Our analysis yields $\mathcal{O}(n^{-1/4})$ convergence rates to the Gaussian limit in the Kolmogorov distance. We further establish the non-asymptotic validity of a multiplier block bootstrap procedure for constructing the confidence intervals, guaranteeing consistent inference under Markovian sampling. Our work provides the first non-asymptotic guarantees on the rate of convergence of bootstrap-based confidence intervals for stochastic approximation with Markov noise. Moreover, we recover the classical rate of order $\mathcal{O}(n^{-1/8})$ up to logarithmic factors for estimating the asymptotic variance of the iterates of the LSA algorithm.