Adaptive Top-K in SGD for Communication-Efficient Distributed Learning

TL;DR

This paper introduces an adaptive Top-K sparsification method for distributed SGD that dynamically adjusts gradient sparsity to improve convergence and communication efficiency, outperforming existing fixed sparsification approaches.

Contribution

It proposes a novel adaptive Top-K sparsification framework with theoretical convergence bounds and an algorithm to optimize sparsification for better training performance.

Findings

Achieves faster convergence rates on MNIST and CIFAR-10 datasets.

Balances communication cost and convergence error effectively.

Outperforms state-of-the-art fixed sparsification methods.

Abstract

Distributed stochastic gradient descent (SGD) with gradient compression has become a popular communication-efficient solution for accelerating distributed learning. One commonly used method for gradient compression is Top-K sparsification, which sparsifies the gradients by a fixed degree during model training. However, there has been a lack of an adaptive approach to adjust the sparsification degree to maximize the potential of the model's performance or training speed. This paper proposes a novel adaptive Top-K in SGD framework that enables an adaptive degree of sparsification for each gradient descent step to optimize the convergence performance by balancing the trade-off between communication cost and convergence error. Firstly, an upper bound of convergence error is derived for the adaptive sparsification scheme and the loss function. Secondly, an algorithm is designed to minimize the convergence error under the communication cost constraints. Finally, numerical results on the MNIST and CIFAR-10 datasets demonstrate that the proposed adaptive Top-K algorithm in SGD achieves a significantly better convergence rate compared to state-of-the-art methods, even after considering error compensation.