CD-SGD: Distributed Stochastic Gradient Descent with Compression and Delay Compensation

TL;DR

This paper introduces CD-SGD, a distributed stochastic gradient descent method that incorporates gradient compression with delay compensation to reduce communication overhead while maintaining convergence accuracy.

Contribution

The paper proposes a novel CD-SGD algorithm that effectively combines gradient compression with delay compensation to improve distributed training efficiency.

Findings

Reduces communication overhead in distributed training

Maintains convergence accuracy despite gradient compression

Demonstrates improved training speed in experiments

Abstract

Communication overhead is the key challenge for distributed training. Gradient compression is a widely used approach to reduce communication traffic. When combining with parallel communication mechanism method like pipeline, gradient compression technique can greatly alleviate the impact of communication overhead. However, there exists two problems of gradient compression technique to be solved. Firstly, gradient compression brings in extra computation cost, which will delay the next training iteration. Secondly, gradient compression usually leads to the decrease of convergence accuracy.