Asynchronous speedup in decentralized optimization

TL;DR

This paper introduces a continuized framework for analyzing asynchronous decentralized optimization algorithms, providing precise convergence characterizations and demonstrating an asynchronous speedup that improves robustness to delays and stragglers.

Contribution

It develops a novel continuized analysis method for asynchronous algorithms in networks with delays, enabling better understanding and improved convergence guarantees.

Findings

Convergence time depends on the eigengap and local delays, not worst-case delays.

Algorithms achieve asynchronous speedup, improving robustness to stragglers.

The framework combines continuous analysis with discrete implementation.

Abstract

In decentralized optimization, nodes of a communication network each possess a local objective function, and communicate using gossip-based methods in order to minimize the average of these per-node functions. While synchronous algorithms are heavily impacted by a few slow nodes or edges in the graph (the \emph{straggler problem}), their asynchronous counterparts are notoriously harder to parametrize. Indeed, their convergence properties for networks with heterogeneous communication and computation delays have defied analysis so far. In this paper, we use a \emph{ continuized} framework to analyze asynchronous algorithms in networks with delays. Our approach yields a precise characterization of convergence time and of its dependency on heterogeneous delays in the network. Our continuized framework benefits from the best of both continuous and discrete worlds: the algorithms it applies to are based on event-driven updates. They are thus essentially discrete and hence readily implementable. Yet their analysis is essentially in continuous time, relying in part on the theory of delayed ODEs. Our algorithms moreover achieve an \emph{asynchronous speedup}: their rate of convergence is controlled by the eigengap of the network graph weighted by local delays, instead of the network-wide worst-case delay as in previous analyses. Our methods thus enjoy improved robustness to stragglers.