Revisiting Distributed Synchronous SGD

TL;DR

This paper reevaluates distributed synchronous SGD, proposing a backup worker approach that reduces noise and improves convergence and accuracy compared to traditional asynchronous and synchronous methods.

Contribution

It introduces a novel backup worker strategy for synchronous SGD that mitigates straggler effects and reduces noise, enhancing training efficiency and model performance.

Findings

Backup worker approach improves convergence speed.

Synchronous with backup workers outperforms traditional methods.

Method achieves better test accuracy.

Abstract

Distributed training of deep learning models on large-scale training data is typically conducted with asynchronous stochastic optimization to maximize the rate of updates, at the cost of additional noise introduced from asynchrony. In contrast, the synchronous approach is often thought to be impractical due to idle time wasted on waiting for straggling workers. We revisit these conventional beliefs in this paper, and examine the weaknesses of both approaches. We demonstrate that a third approach, synchronous optimization with backup workers, can avoid asynchronous noise while mitigating for the worst stragglers. Our approach is empirically validated and shown to converge faster and to better test accuracies.