High-Order Error Bounds for Markovian LSA with Richardson-Romberg Extrapolation

TL;DR

This paper investigates the bias and error bounds of Markovian LSA with PR averaging, introducing Richardson-Romberg extrapolation to cancel leading bias and improve high-order error estimates.

Contribution

It presents a novel bias decomposition for LSA under Markovian noise and applies Richardson-Romberg extrapolation to achieve high-order error bounds.

Findings

Richardson-Romberg effectively cancels the leading bias term.

High-order moment bounds are derived for the RR iterates.

Leading error aligns with the optimal covariance of vanilla LSA.

Abstract

In this paper, we study the bias and high-order error bounds of the Linear Stochastic Approximation (LSA) algorithm with Polyak-Ruppert (PR) averaging under Markovian noise. We focus on the version of the algorithm with constant step size $\alpha$ and propose a novel decomposition of the bias via a linearization technique. We analyze the structure of the bias and show that the leading-order term is linear in $\alpha$ and cannot be eliminated by PR averaging. To address this, we apply the Richardson-Romberg (RR) extrapolation procedure, which effectively cancels the leading bias term. We derive high-order moment bounds for the RR iterates and show that the leading error term aligns with the asymptotically optimal covariance matrix of the vanilla averaged LSA iterates.