Empirical Analysis on Top-k Gradient Sparsification for Distributed Deep Learning in a Supercomputing Environment

TL;DR

This paper empirically analyzes the inefficiencies of Top-k gradient sparsification in distributed deep learning on GPUs, highlighting communication bottlenecks and suggesting directions for more efficient methods.

Contribution

The paper provides an empirical evaluation of Top-k SGD's performance limitations on GPUs, offering insights for developing more efficient gradient sparsification techniques.

Findings

Top-k SGD is inefficient due to gradient sorting on GPUs.

Gradient sparsification reduces communication but has performance trade-offs.

Empirical analysis reveals bottlenecks in current sparsification methods.

Abstract

To train deep learning models faster, distributed training on multiple GPUs is the very popular scheme in recent years. However, the communication bandwidth is still a major bottleneck of training performance. To improve overall training performance, recent works have proposed gradient sparsification methods that reduce the communication traffic significantly. Most of them require gradient sorting to select meaningful gradients such as Top-k gradient sparsification (Top-k SGD). However, Top-k SGD has a limit to increase the speed up overall training performance because gradient sorting is significantly inefficient on GPUs. In this paper, we conduct experiments that show the inefficiency of Top-k SGD and provide the insight of the low performance. Based on observations from our empirical analysis, we plan to yield a high performance gradient sparsification method as a future work.