Understanding Top-k Sparsification in Distributed Deep Learning

TL;DR

This paper investigates the behavior of Top-k sparsification in distributed deep learning, providing a tighter theoretical analysis, empirical insights into gradient distributions, and an efficient approximate top-k selection algorithm to enhance scalability.

Contribution

It offers a detailed analysis of Top-k sparsification, derives a tighter convergence bound, and proposes a GPU-efficient approximate top-k algorithm for improved scalability.

Findings

Gradient distributions during training are characterized.

A tighter convergence bound for TopK-SGD is derived.

An efficient approximate top-k algorithm reduces computational overhead.

Abstract

Distributed stochastic gradient descent (SGD) algorithms are widely deployed in training large-scale deep learning models, while the communication overhead among workers becomes the new system bottleneck. Recently proposed gradient sparsification techniques, especially Top-$k$ sparsification with error compensation (TopK-SGD), can significantly reduce the communication traffic without an obvious impact on the model accuracy. Some theoretical studies have been carried out to analyze the convergence property of TopK-SGD. However, existing studies do not dive into the details of Top-$k$ operator in gradient sparsification and use relaxed bounds (e.g., exact bound of Random-$k$) for analysis; hence the derived results cannot well describe the real convergence performance of TopK-SGD. To this end, we first study the gradient distributions of TopK-SGD during the training process through extensive experiments. We then theoretically derive a tighter bound for the Top-$k$ operator. Finally, we exploit the property of gradient distribution to propose an approximate top-$k$ selection algorithm, which is computing-efficient for GPUs, to improve the scaling efficiency of TopK-SGD by significantly reducing the computing overhead. Codes are available at: \url{https://github.com/hclhkbu/GaussianK-SGD}.