Robust Fully-Asynchronous Methods for Distributed Training over General Architecture

TL;DR

This paper introduces R-FAST, a robust asynchronous distributed training method that handles data heterogeneity, packet loss, and flexible communication architectures, achieving faster convergence and comparable accuracy to synchronous methods.

Contribution

The paper proposes R-FAST, a novel asynchronous distributed training algorithm with robust gradient tracking and flexible communication, improving efficiency and resilience over existing methods.

Findings

R-FAST converges to a neighborhood of the optimum with geometric rate for convex problems.

R-FAST converges to a stationary point with sublinear rate for non-convex problems.

R-FAST is 1.5-2 times faster than synchronous benchmarks and outperforms existing asynchronous algorithms.

Abstract

Perfect synchronization in distributed machine learning problems is inefficient and even impossible due to the existence of latency, package losses and stragglers. We propose a Robust Fully-Asynchronous Stochastic Gradient Tracking method (R-FAST), where each device performs local computation and communication at its own pace without any form of synchronization. Different from existing asynchronous distributed algorithms, R-FAST can eliminate the impact of data heterogeneity across devices and allow for packet losses by employing a robust gradient tracking strategy that relies on properly designed auxiliary variables for tracking and buffering the overall gradient vector. More importantly, the proposed method utilizes two spanning-tree graphs for communication so long as both share at least one common root, enabling flexible designs in communication architectures. We show that R-FAST converges in expectation to a neighborhood of the optimum with a geometric rate for smooth and strongly convex objectives; and to a stationary point with a sublinear rate for general non-convex settings. Extensive experiments demonstrate that R-FAST runs 1.5-2 times faster than synchronous benchmark algorithms, such as Ring-AllReduce and D-PSGD, while still achieving comparable accuracy, and outperforms existing asynchronous SOTA algorithms, such as AD-PSGD and OSGP, especially in the presence of stragglers.