Asynchronous Iterations in Optimization: New Sequence Results and Sharper Algorithmic Guarantees

TL;DR

This paper presents new convergence results for asynchronous optimization algorithms, providing explicit bounds on how delays affect convergence and improving theoretical guarantees for several methods.

Contribution

It introduces simplified, stronger convergence results for asynchronous iterations, enabling better analysis and guarantees for multiple optimization algorithms.

Findings

Derived improved iteration complexity bounds for proximal incremental gradient methods.

Provided tighter convergence guarantees for asynchronous stochastic gradient descent.

Quantified convergence rates for totally asynchronous iterations under various delay assumptions.

Abstract

We introduce novel convergence results for asynchronous iterations that appear in the analysis of parallel and distributed optimization algorithms. The results are simple to apply and give explicit estimates for how the degree of asynchrony impacts the convergence rates of the iterates. Our results shorten, streamline and strengthen existing convergence proofs for several asynchronous optimization methods and allow us to establish convergence guarantees for popular algorithms that were thus far lacking a complete theoretical understanding. Specifically, we use our results to derive better iteration complexity bounds for proximal incremental aggregated gradient methods, to obtain tighter guarantees depending on the average rather than maximum delay for the asynchronous stochastic gradient descent method, to provide less conservative analyses of the speedup conditions for asynchronous block-coordinate implementations of Krasnoselskii-Mann iterations, and to quantify the convergence rates for totally asynchronous iterations under various assumptions on communication delays and update rates.