Stochastic Krasnoselskii-Mann Iterations: Convergence without Uniformly Bounded Variance

TL;DR

This paper proves convergence of stochastic fixed-point iterations under weaker variance assumptions, extending guarantees to algorithms like stochastic gradient descent and splitting methods.

Contribution

It establishes convergence without requiring uniformly bounded variance, a significant relaxation of typical assumptions in stochastic fixed-point algorithms.

Findings

Almost sure weak convergence of iterates

Convergence rates for expected residuals

First results for stochastic splitting methods without uniform variance bounds

Abstract

We investigate the Stochastic Krasnoselskii-Mann iterations for expected nonexpansive fixed-point problems in a real Hilbert space. We establish convergence guarantees under significantly weaker assumptions on the variance than those typically used in the literature. In particular, instead of a uniform bound on the variance of the stochastic oracle, we only assume finite variance at a single fixed point. Under this assumption, we prove almost sure weak convergence of the iterates, derive convergence rates for the expected residual, and obtain almost sure convergence rates for the running minimum residual. Notably, we recover the best-known stochastic oracle complexity without imposing uniformly bounded variance. We illustrate the applicability of our results to Stochastic Gradient Descent, where we recover known guarantees, and to Stochastic Three-Operator Splitting and Stochastic Backward-Forward Splitting, for which we obtain the first results that avoid uniform variance bounds.