Heavy-Tailed and Long-Range Dependent Noise in Stochastic Approximation: A Finite-Time Analysis

TL;DR

This paper develops finite-time convergence bounds for stochastic approximation algorithms under heavy-tailed and long-range dependent noise, extending classical results to more realistic noise models in reinforcement learning and optimization.

Contribution

It introduces the first finite-time moment bounds for SA with heavy-tailed and LRD noise, using a noise-averaging approach that does not alter the iteration.

Findings

Finite-time bounds for heavy-tailed noise in SA.

Finite-time bounds for long-range dependent noise in SA.

Numerical experiments validating the theoretical results.

Abstract

Stochastic approximation (SA) is a fundamental iterative framework with broad applications in reinforcement learning and optimization. Classical analyses typically rely on martingale difference or Markov noise with bounded second moments, but many practical settings, including finance and communications, frequently encounter heavy-tailed and long-range dependent (LRD) noise. In this work, we study SA for finding the root of a strongly monotone operator under these non-classical noise models. We establish the first finite-time moment bounds in both settings, providing explicit convergence rates that quantify the impact of heavy tails and temporal dependence. Our analysis employs a noise-averaging argument that regularizes the impact of noise without modifying the iteration. Finally, we apply our general framework to stochastic gradient descent (SGD) and gradient play, and corroborate our finite-time analysis through numerical experiments.