GraphPipe: Improving Performance and Scalability of DNN Training with Graph Pipeline Parallelism

TL;DR

GraphPipe introduces graph pipeline parallelism (GPP), a novel approach that leverages DNN topology to improve training performance and scalability by enabling concurrent execution of independent operators.

Contribution

The paper proposes GPP, a new pipeline-parallel scheme that preserves DNN topology for better concurrency, and develops GraphPipe, a system that implements GPP for scalable DNN training.

Findings

GraphPipe outperforms PipeDream and Piper by up to 1.6X in training speed.

GraphPipe reduces search time by 9-21X compared to existing systems.

GPP enables concurrent execution of independent operators, improving GPU performance.

Abstract

Deep neural networks (DNNs) continue to grow rapidly in size, making them infeasible to train on a single device. Pipeline parallelism is commonly used in existing DNN systems to support large-scale DNN training by partitioning a DNN into multiple stages, which concurrently perform DNN training for different micro-batches in a pipeline fashion. However, existing pipeline-parallel approaches only consider sequential pipeline stages and thus ignore the topology of a DNN, resulting in missed model-parallel opportunities. This paper presents graph pipeline parallelism (GPP), a new pipeline-parallel scheme that partitions a DNN into pipeline stages whose dependencies are identified by a directed acyclic graph. GPP generalizes existing sequential pipeline parallelism and preserves the inherent topology of a DNN to enable concurrent execution of computationally-independent operators, resulting in reduced memory requirement and improved GPU performance. In addition, we develop GraphPipe, a distributed system that exploits GPP strategies to enable performant and scalable DNN training. GraphPipe partitions a DNN into a graph of stages, optimizes micro-batch schedules for these stages, and parallelizes DNN training using the discovered GPP strategies. Evaluation on a variety of DNNs shows that GraphPipe outperforms existing pipeline-parallel systems such as PipeDream and Piper by up to 1.6X. GraphPipe also reduces the search time by 9-21X compared to PipeDream and Piper.