Shuffle-Exchange Brings Faster: Reduce the Idle Time During Communication for Decentralized Neural Network Training

TL;DR

This paper introduces Shuffle-Exchange SGD (SESGD), a novel communication scheme that significantly reduces idle time during distributed neural network training, achieving up to 5x speedup in high latency networks.

Contribution

SESGD replaces Ring-AllReduce with a shuffle-exchange communication pattern, reducing handshake overhead and improving training speed in distributed deep learning.

Findings

SESGD achieves 1.7x speedup in low latency networks.

SESGD achieves up to 5x speedup in high latency networks.

Model accuracy remains unaffected by SESGD.

Abstract

As a crucial scheme to accelerate the deep neural network (DNN) training, distributed stochastic gradient descent (DSGD) is widely adopted in many real-world applications. In most distributed deep learning (DL) frameworks, DSGD is implemented with Ring-AllReduce architecture (Ring-SGD) and uses a computation-communication overlap strategy to address the overhead of the massive communications required by DSGD. However, we observe that although $O(1)$ gradients are needed to be communicated per worker in Ring-SGD, the $O(n)$ handshakes required by Ring-SGD limits its usage when training with many workers or in high latency network. In this paper, we propose Shuffle-Exchange SGD (SESGD) to solve the dilemma of Ring-SGD. In the cluster of 16 workers with 0.1ms Ethernet latency, SESGD can accelerate the DNN training to $1.7 \times$ without losing model accuracy. Moreover, the process can be accelerated up to $5\times$ in high latency networks (5ms).