Robust Asynchronous Stochastic Gradient-Push: Asymptotically Optimal and Network-Independent Performance for Strongly Convex Functions

TL;DR

This paper introduces a robust distributed optimization method that performs optimally in harsh network conditions with asynchronous updates, message delays, and losses, specifically for strongly convex functions.

Contribution

It proposes a modified Gradient-Push algorithm that achieves asymptotically optimal performance under challenging network scenarios with noisy gradient information.

Findings

Achieves asymptotic performance comparable to centralized gradient descent.

Handles asynchronous updates, message delays, and message losses effectively.

Works for strongly convex functions with Lipschitz gradients.

Abstract

We consider the standard model of distributed optimization of a sum of functions $F(\bz) = \sum_{i=1}^n f_i(\bz)$, where node $i$ in a network holds the function $f_i(\bz)$. We allow for a harsh network model characterized by asynchronous updates, message delays, unpredictable message losses, and directed communication among nodes. In this setting, we analyze a modification of the Gradient-Push method for distributed optimization, assuming that \begin{enumerate*}[label=(\roman*)] \item node $i$ is capable of generating gradients of its function $f_i(\bz)$ corrupted by zero-mean bounded-support additive noise at each step, \item $F(\bz)$ is strongly convex, and \item each $f_i(\bz)$ has Lipschitz gradients. We show that our proposed method asymptotically performs as well as the best bounds on centralized gradient descent that takes steps in the direction of the sum of the noisy gradients of all the functions $f_1(\bz), \ldots, f_n(\bz)$ at each step.