Distributed SGD Generalizes Well Under Asynchrony

TL;DR

This paper analyzes how asynchronous distributed stochastic gradient descent (SGD) maintains good generalization performance despite communication delays, and proposes adaptive learning rates to enhance stability and reduce errors.

Contribution

It provides a theoretical analysis of the generalization ability of asynchronous distributed SGD and introduces an adaptive learning rate strategy to improve stability.

Findings

Asynchrony can be managed with appropriate learning rate adjustments.

Distributed SGD generalizes well with sufficient data samples.

Adaptive learning rates improve stability and reduce generalization error.

Abstract

The performance of fully synchronized distributed systems has faced a bottleneck due to the big data trend, under which asynchronous distributed systems are becoming a major popularity due to their powerful scalability. In this paper, we study the generalization performance of stochastic gradient descent (SGD) on a distributed asynchronous system. The system consists of multiple worker machines that compute stochastic gradients which are further sent to and aggregated on a common parameter server to update the variables, and the communication in the system suffers from possible delays. Under the algorithm stability framework, we prove that distributed asynchronous SGD generalizes well given enough data samples in the training optimization. In particular, our results suggest to reduce the learning rate as we allow more asynchrony in the distributed system. Such adaptive learning rate strategy improves the stability of the distributed algorithm and reduces the corresponding generalization error. Then, we confirm our theoretical findings via numerical experiments.