Distributed Training of Deep Learning Models: A Taxonomic Perspective

TL;DR

This paper provides a comprehensive taxonomy of distributed deep learning systems, analyzing their fundamental principles, techniques, and implications to facilitate understanding and comparison of various approaches in distributed training.

Contribution

It introduces a taxonomy categorizing distributed deep learning systems based on their techniques and principles, aiding in systematic analysis and comparison.

Findings

Identifies key principles underlying distributed deep learning

Categorizes existing techniques into a coherent taxonomy

Highlights implications of different approaches on training efficiency

Abstract

Distributed deep learning systems (DDLS) train deep neural network models by utilizing the distributed resources of a cluster. Developers of DDLS are required to make many decisions to process their particular workloads in their chosen environment efficiently. The advent of GPU-based deep learning, the ever-increasing size of datasets and deep neural network models, in combination with the bandwidth constraints that exist in cluster environments require developers of DDLS to be innovative in order to train high quality models quickly. Comparing DDLS side-by-side is difficult due to their extensive feature lists and architectural deviations. We aim to shine some light on the fundamental principles that are at work when training deep neural networks in a cluster of independent machines by analyzing the general properties associated with training deep learning models and how such workloads can be distributed in a cluster to achieve collaborative model training. Thereby we provide an overview of the different techniques that are used by contemporary DDLS and discuss their influence and implications on the training process. To conceptualize and compare DDLS, we group different techniques into categories, thus establishing a taxonomy of distributed deep learning systems.