XPipe: Efficient Pipeline Model Parallelism for Multi-GPU DNN Training

TL;DR

XPipe introduces an asynchronous pipeline model parallelism method for multi-GPU DNN training that improves GPU utilization and throughput while maintaining model accuracy through a novel weight prediction strategy.

Contribution

It presents XPipe, a new asynchronous pipeline parallelism approach with a weight prediction technique that balances efficiency and accuracy in multi-GPU training.

Findings

XPipe achieves higher throughput than existing methods.

XPipe maintains comparable or better model accuracy.

Experimental results outperform state-of-the-art approaches.

Abstract

We propose XPipe, an efficient asynchronous pipeline model parallelism approach for multi-GPU DNN training. XPipe is designed to use multiple GPUs to concurrently and continuously train different parts of a DNN model. To improve GPU utilization and achieve high throughput, it splits a mini-batch into a set of micro-batches. It allows the overlapping of the pipelines of multiple micro-batches, including those belonging to different mini-batches. Most importantly, the novel weight prediction strategy adopted by XPipe enables it to effectively address the weight inconsistency and staleness issues incurred by the asynchronous pipeline parallelism. As a result, XPipe incorporates the advantages of both synchronous and asynchronous pipeline model parallelism approaches. Concretely, it can achieve very comparable (even slightly better) model accuracy as its synchronous counterpart while obtaining higher throughput than it. Experimental results show that XPipe outperforms other state-of-the-art synchronous and asynchronous model parallelism approaches.