A Class of Accelerated Fixed-Point-Based Methods with Delayed Inexact Oracles and Its Applications

TL;DR

This paper introduces a new accelerated fixed-point method that handles delayed inexact oracles, achieving faster convergence rates and broad applicability to asynchronous and stochastic settings in scientific computing.

Contribution

It develops a unified accelerated framework with delayed inexact oracles, providing improved convergence rates and applicability to various practical scenarios including stochastic and asynchronous algorithms.

Findings

Achieves $oldsymbol{O(1/k^2)}$ convergence rate for residuals.

Proves almost sure convergence of iterates.

Demonstrates effectiveness through numerical experiments on matrix games and neural networks.

Abstract

In this paper, we develop a novel accelerated fixed-point-based framework using delayed inexact oracles to approximate a fixed point of a nonexpansive operator (or equivalently, a root of a co-coercive operator), a central problem in scientific computing. Our approach leverages both Nesterov's acceleration technique and the Krasnosel'skii-Mann (KM) iteration, while accounting for delayed inexact oracles, a key mechanism in asynchronous algorithms. We also introduce a unified approximate error condition for delayed inexact oracles, which can cover various practical scenarios. Under mild conditions and appropriate parameter updates, we establish both $\mathcal{O}(1/k^2)$ non-asymptotic and $o(1/k^2)$ asymptotic convergence rates in expectation for the squared norm of residual. Our rate significantly improves the $\mathcal{O}(1/k)$ rates in classical KM-type methods, including their asynchronous variants. We also establish $o(1/k^2)$ almost sure convergence rates and the almost sure convergence of iterates to a solution of the problem. Within our framework, we instantiate three settings for the underlying operator: (i) a deterministic universal delayed oracle; (ii) a stochastic delayed oracle; and (iii) a finite-sum structure with asynchronous updates. For each case, we instantiate our framework to obtain a concrete algorithmic variant for which our convergence results still apply, and whose iteration complexity depends linearly on the maximum delay. Finally, we verify our algorithms and theoretical results through two numerical examples on both matrix game and shallow neural network training problems.