Asymptotic regularity of a generalised stochastic Halpern scheme

TL;DR

This paper establishes uniform asymptotic regularity rates for a broad class of stochastic iterative schemes, including variants of Halpern and Krasnoselskii-Mann iterations, with applications to optimization and reinforcement learning.

Contribution

It introduces a highly general stochastic iteration framework that unifies and extends existing schemes, providing the first quadratic rates in inner product spaces and linear rates for specific stochastic optimization methods.

Findings

Linear rates of asymptotic regularity for stochastic schemes

Quadratic convergence rates in inner product spaces

Extension to stochastic variants of Q-learning in reinforcement learning

Abstract

We provide abstract, general and highly uniform rates of asymptotic regularity for a generalized stochastic Halpern-style iteration, which incorporates a second mapping in the style of a Krasnoselskii-Mann iteration. This iteration is general in two ways: First, it incorporates stochasticity completely abstractly, rather than fixing a sampling method; second, it includes as special cases stochastic versions of various schemes from the optimization literature, including Halpern's iteration as well as a Krasnoselskii-Mann iteration with Tikhonov regularization terms in the sense of Bo\c{t}, Csetnek and Meier (where this stochastic variant of the latter is considered for the first time in this paper). For these specific cases, we obtain linear rates of asymptotic regularity, matching (or improving) the currently best known rates for these iterations in stochastic optimization, and quadratic rates of asymptotic regularity are obtained in the context of inner product spaces for the general iteration. We conclude by discussing how variance can be managed in practice through sampling methods in the style of minibatching, how our convergence rates can be adapted to provide oracle complexity bounds, and by sketching how the schemes presented here can be instantiated in the context of reinforcement learning to yield novel methods for Q-learning.