Priority-based Parameter Propagation for Distributed DNN Training

TL;DR

This paper introduces Priority-based Parameter Propagation (P3), a novel synchronization method that overlaps communication with computation in distributed DNN training, significantly improving throughput by up to 66%.

Contribution

The paper proposes P3, a new synchronization mechanism that uses finer granularity and scheduling to reduce communication delays in distributed DNN training.

Findings

P3 improves training throughput by up to 66%.

P3 effectively overlaps communication with computation.

Different parameters can tolerate different synchronization delays.

Abstract

Data parallel training is widely used for scaling distributed deep neural network (DNN) training. However, the performance benefits are often limited by the communication-heavy parameter synchronization step. In this paper, we take advantage of the domain specific knowledge of DNN training and overlap parameter synchronization with computation in order to improve the training performance. We make two key observations: (1) the optimal data representation granularity for the communication may differ from that used by the underlying DNN model implementation and (2) different parameters can afford different synchronization delays. Based on these observations, we propose a new synchronization mechanism called Priority-based Parameter Propagation (P3). P3 synchronizes parameters at a finer granularity and schedules data transmission in such a way that the training process incurs minimal communication delay. We show that P3 can improve the training throughput of ResNet-50, Sockeye and VGG-19 by as much as 25%, 38% and 66% respectively on clusters with realistic network bandwidth