OSP: Boosting Distributed Model Training with 2-stage Synchronization

TL;DR

This paper introduces OSP, a novel 2-stage synchronization method with local-gradient correction for distributed deep learning, significantly improving throughput while maintaining accuracy.

Contribution

The paper proposes OSP, a new synchronization approach combining 2-stage communication and local-gradient correction to enhance efficiency and accuracy in distributed training.

Findings

Up to 50% throughput improvement over existing methods

Maintains accuracy without loss due to stale parameters

Effective on multiple deep learning models and datasets

Abstract

Distributed deep learning (DDL) is a promising research area, which aims to increase the efficiency of training deep learning tasks with large size of datasets and models. As the computation capability of DDL nodes continues to increase, the network connection between nodes is becoming a major bottleneck. Various methods of gradient compression and improved model synchronization have been proposed to address this bottleneck in Parameter-Server-based DDL. However, these two types of methods can result in accuracy loss due to discarded gradients and have limited enhancement on the throughput of model synchronization, respectively. To address these challenges, we propose a new model synchronization method named Overlapped Synchronization Parallel (OSP), which achieves efficient communication with a 2-stage synchronization approach and uses Local-Gradient-based Parameter correction (LGP) to avoid accuracy loss caused by stale parameters. The prototype of OSP has been implemented using PyTorch and evaluated on commonly used deep learning models and datasets with a 9-node testbed. Evaluation results show that OSP can achieve up to 50\% improvement in throughput without accuracy loss compared to popular synchronization models.