Stochastic fixed-point iterations for nonexpansive maps: Convergence and error bounds

TL;DR

This paper analyzes a stochastic version of the Krasnoselski--Mann iteration for finding fixed points of nonexpansive maps, providing convergence guarantees and error bounds under various noise conditions, with applications to reinforcement learning.

Contribution

It introduces convergence analysis and error bounds for a stochastic fixed-point iteration under unbounded and bounded noise, extending prior deterministic results.

Findings

Almost sure convergence under martingale difference noise

Explicit error bounds with computable constants

Application to reinforcement learning for Markov decision processes

Abstract

We study a stochastically perturbed version of the well-known Krasnoselski--Mann iteration for computing fixed points of nonexpansive maps in finite dimensional normed spaces. We discuss sufficient conditions on the stochastic noise and stepsizes that guarantee almost sure convergence of the iterates towards a fixed point, and derive non-asymptotic error bounds and convergence rates for the fixed-point residuals. Our main results concern the case of a martingale difference noise with variances that can possibly grow unbounded. This supports an application to reinforcement learning for average reward Markov decision processes, for which we establish convergence and asymptotic rates. We also analyze in depth the case where the noise has uniformly bounded variance, obtaining error bounds with explicit computable constants.