DEFT: Exploiting Gradient Norm Difference between Model Layers for Scalable Gradient Sparsification

TL;DR

DEFT is a scalable gradient sparsification method for distributed deep learning that partitions gradient selection tasks among workers, reducing computational costs and communication traffic while maintaining high convergence.

Contribution

DEFT introduces a novel partitioned gradient selection scheme that improves scalability and efficiency in distributed training.

Findings

Significant speedup in gradient selection compared to existing methods.

Maintains high convergence performance.

Reduces communication traffic regardless of the number of workers.

Abstract

Gradient sparsification is a widely adopted solution for reducing the excessive communication traffic in distributed deep learning. However, most existing gradient sparsifiers have relatively poor scalability because of considerable computational cost of gradient selection and/or increased communication traffic owing to gradient build-up. To address these challenges, we propose a novel gradient sparsification scheme, DEFT, that partitions the gradient selection task into sub tasks and distributes them to workers. DEFT differs from existing sparsifiers, wherein every worker selects gradients among all gradients. Consequently, the computational cost can be reduced as the number of workers increases. Moreover, gradient build-up can be eliminated because DEFT allows workers to select gradients in partitions that are non-intersecting (between workers). Therefore, even if the number of workers increases, the communication traffic can be maintained as per user requirement. To avoid the loss of significance of gradient selection, DEFT selects more gradients in the layers that have a larger gradient norm than the other layers. Because every layer has a different computational load, DEFT allocates layers to workers using a bin-packing algorithm to maintain a balanced load of gradient selection between workers. In our empirical evaluation, DEFT shows a significant improvement in training performance in terms of speed in gradient selection over existing sparsifiers while achieving high convergence performance.