Distributed Deep Learning Using Synchronous Stochastic Gradient Descent

TL;DR

This paper presents a distributed synchronous SGD algorithm for deep learning that achieves near-linear scaling on multiple nodes, enabling faster training of CNNs and DNNs without changing hyperparameters or data compression.

Contribution

The authors introduce a scalable distributed synchronous SGD method that maintains algorithmic integrity and demonstrates record training throughput on large clusters.

Findings

90X scaling of VGG-A on 128 nodes

53X and 42X scaling of VGG-A and OverFeat-FAST on 64 nodes

14X scaling on an Ethernet-based AWS cluster

Abstract

We design and implement a distributed multinode synchronous SGD algorithm, without altering hyper parameters, or compressing data, or altering algorithmic behavior. We perform a detailed analysis of scaling, and identify optimal design points for different networks. We demonstrate scaling of CNNs on 100s of nodes, and present what we believe to be record training throughputs. A 512 minibatch VGG-A CNN training run is scaled 90X on 128 nodes. Also 256 minibatch VGG-A and OverFeat-FAST networks are scaled 53X and 42X respectively on a 64 node cluster. We also demonstrate the generality of our approach via best-in-class 6.5X scaling for a 7-layer DNN on 16 nodes. Thereafter we attempt to democratize deep-learning by training on an Ethernet based AWS cluster and show ~14X scaling on 16 nodes.