Accelerating Distributed ML Training via Selective Synchronization

TL;DR

This paper introduces exttt{SelSync}, a dynamic synchronization method for distributed deep neural network training that reduces communication overhead while maintaining or improving accuracy, achieving up to 14× faster training.

Contribution

exttt{SelSync} is a novel, low-overhead, semi-synchronous training approach that adaptively chooses synchronization steps based on update significance.

Findings

Converges to same or better accuracy than BSP.

Reduces training time by up to 14×.

Effective in semi-synchronous training scenarios.

Abstract

In distributed training, deep neural networks (DNNs) are launched over multiple workers concurrently and aggregate their local updates on each step in bulk-synchronous parallel (BSP) training. However, BSP does not linearly scale-out due to high communication cost of aggregation. To mitigate this overhead, alternatives like Federated Averaging (FedAvg) and Stale-Synchronous Parallel (SSP) either reduce synchronization frequency or eliminate it altogether, usually at the cost of lower final accuracy. In this paper, we present \texttt{SelSync}, a practical, low-overhead method for DNN training that dynamically chooses to incur or avoid communication at each step either by calling the aggregation op or applying local updates based on their significance. We propose various optimizations as part of \texttt{SelSync} to improve convergence in the context of \textit{semi-synchronous} training. Our system converges to the same or better accuracy than BSP while reducing training time by up to 14$\times$.