On Unbounded Delays in Asynchronous Parallel Fixed-Point Algorithms

TL;DR

This paper proves convergence of the ARock asynchronous parallel algorithm under unbounded delays, broadening its applicability and providing more practical step size strategies for large-scale optimization tasks.

Contribution

It establishes convergence of ARock with unbounded delays, introduces adaptive step sizes, and develops new Lyapunov functions for analysis, expanding the algorithm's practical scope.

Findings

Convergence is proven under stochastic and deterministic unbounded delays.

Adaptive step sizes are larger and more practical than previous bounds.

Applicable to large-scale machine learning and scientific computing algorithms.

Abstract

The need for scalable numerical solutions has motivated the development of asynchronous parallel algorithms, where a set of nodes run in parallel with little or no synchronization, thus computing with delayed information. This paper studies the convergence of the asynchronous parallel algorithm ARock under potentially unbounded delays. ARock is a general asynchronous algorithm that has many applications. It parallelizes fixed-point iterations by letting a set of nodes randomly choose solution coordinates and update them in an asynchronous parallel fashion. ARock takes some recent asynchronous coordinate descent algorithms as special cases and gives rise to new asynchronous operator-splitting algorithms. Existing analysis of ARock assumes the delays to be bounded, and uses this bound to set a step size that is important to both convergence and efficiency. Other work, though allowing unbounded delays, imposes strict conditions on the underlying fixed-point operator, resulting in limited applications. In this paper, convergence is established under unbounded delays, which can be either stochastic or deterministic. The proposed step sizes are more practical and generally larger than those in the existing work. The step size adapts to the delay distribution or the current delay being experienced in the system. New Lyapunov functions, which are the key to analyzing asynchronous algorithms, are generated to obtain our results. A set of applicable optimization algorithms with large-scale applications are given, including machine learning and scientific computing algorithms.