A Quantitative Survey of Communication Optimizations in Distributed Deep Learning

TL;DR

This paper provides a comprehensive quantitative survey of communication optimization techniques in distributed deep learning, analyzing their effectiveness and limitations across different system levels and models.

Contribution

It classifies existing communication optimization solutions into three levels and compares seven distributed DL methods on a high-performance GPU cluster.

Findings

Low model intensity models like BERT are hard to scale even with advanced algorithms.

System architecture and scheduling significantly affect scalability.

Lossless algorithms have limitations at high bandwidths for certain models.

Abstract

Nowadays, large and complex deep learning (DL) models are increasingly trained in a distributed manner across multiple worker machines, in which extensive communications between workers pose serious scaling problems. In this article, we present a quantitative survey of communication optimization techniques for data parallel distributed DL. We first identify the major communication challenges and classify the existing solutions into three levels, namely the learning algorithm, the system architecture, and the network infrastructure. We present the state-of-the-art communication optimization techniques and conduct a comparative study of seven common lossless distributed DL methods on a 32-GPU cluster with 100Gbps InfiniBand (IB). We show that (1) the DL models with low model intensity (such as BERT and BERT-Large) are difficult to scale out even with the best available lossless algorithm over 100Gbps IB; (2) the system architecture and scheduling algorithms have a critical impact on the scaling property. We conclude the article with discussions on the open issues for further investigations.