HPSGD: Hierarchical Parallel SGD With Stale Gradients Featuring

TL;DR

HPSGD introduces a hierarchical parallel training strategy that overlaps synchronization with local training and uses an improved update method to mitigate stale gradients, significantly enhancing distributed DNN training efficiency and accuracy.

Contribution

The paper proposes a novel hierarchical parallel SGD method that overlaps synchronization with local training and employs an improved update to address stale gradients.

Findings

Substantially boosts distributed DNN training speed

Reduces the impact of stale gradients

Achieves better accuracy within fixed training time

Abstract

While distributed training significantly speeds up the training process of the deep neural network (DNN), the utilization of the cluster is relatively low due to the time-consuming data synchronizing between workers. To alleviate this problem, a novel Hierarchical Parallel SGD (HPSGD) strategy is proposed based on the observation that the data synchronization phase can be paralleled with the local training phase (i.e., Feed-forward and back-propagation). Furthermore, an improved model updating method is unitized to remedy the introduced stale gradients problem, which commits updates to the replica (i.e., a temporary model that has the same parameters as the global model) and then merges the average changes to the global model. Extensive experiments are conducted to demonstrate that the proposed HPSGD approach substantially boosts the distributed DNN training, reduces the disturbance of the stale gradients and achieves better accuracy in given fixed wall-time.