Concentration of General Stochastic Approximation Under Heavy-Tailed Markovian Noise

TL;DR

This paper derives maximal concentration bounds for stochastic approximation algorithms with Markovian noise, revealing how tail behaviors depend on step sizes and operator properties, with implications for error tail heaviness.

Contribution

It introduces a novel Lyapunov function and truncation technique to analyze concentration bounds under general step sizes and Markovian noise in stochastic approximation.

Findings

Error tail can be sub-Gaussian, sub-Weibull, or heavier depending on conditions.

Sharp bounds are proven to be impossible through worst-case examples.

Error tail heaviness relates to the operator being non-expansive or expansive.

Abstract

We establish maximal concentration bounds for the iterates generated by stochastic approximation algorithms with general step sizes, where the noise has a finite-state Markovian component plus a Martingale-difference component. When the Martingale-difference noise is bounded, we show that the tail of the error can be sub-Gaussian, sub-Weibull, or something lighter than any Pareto but heavier than any Weibull, depending on the step size sequence and on whether the random operator is almost surely contractive, almost surely non-expansive, or expansive with positive probability. Our analysis relies on a novel Lyapunov function involving the moment-generating function of the solution to a Poisson equation, together with an auxiliary projected algorithm. We complement the upper bounds with worst-case examples showing that qualitatively sharper bounds are impossible. We further study the case of unbounded Martingale-difference noise when the average operator is contractive, and the step sizes are of order $1/k$. In this setting, we show that if the random operator is almost surely non-expansive, then the error tail is at most three times heavier than the noise tail, whereas if the random operator is expansive with positive probability, then the error may have substantially heavier tails. These results are obtained through a novel black-box truncation argument that reduces the unbounded-noise setting to the bounded-noise case.