Faster Multi-GPU Training with PPLL: A Pipeline Parallelism Framework Leveraging Local Learning

TL;DR

This paper introduces PPLL, a pipeline parallelism framework based on local learning, which improves multi-GPU training efficiency by reducing communication overhead and synchronization delays.

Contribution

PPLL leverages local learning and queue-based data transfer to enable seamless, efficient pipeline parallelism across multiple GPUs, outperforming traditional methods.

Findings

PPLL accelerates ViT training by 162% on 4 GPUs.

ResNet training speed increased by 33% with PPLL.

PPLL achieves comparable or better speed than traditional pipeline parallelism.

Abstract

Currently, training large-scale deep learning models is typically achieved through parallel training across multiple GPUs. However, due to the inherent communication overhead and synchronization delays in traditional model parallelism methods, seamless parallel training cannot be achieved, which, to some extent, affects overall training efficiency. To address this issue, we present PPLL (Pipeline Parallelism based on Local Learning), a novel framework that leverages local learning algorithms to enable effective parallel training across multiple GPUs. PPLL divides the model into several distinct blocks, each allocated to a separate GPU. By utilizing queues to manage data transfers between GPUs, PPLL ensures seamless cross-GPU communication, allowing multiple blocks to execute forward and backward passes in a pipelined manner. This design minimizes idle times and prevents bottlenecks typically caused by sequential gradient updates, thereby accelerating the overall training process. We validate PPLL through extensive experiments using ResNet and Vision Transformer (ViT) architectures on CIFAR-10, SVHN, and STL-10 datasets. Our results demonstrate that PPLL significantly enhances the training speed of the local learning method while achieving comparable or even superior training speed to traditional pipeline parallelism (PP) without sacrificing model performance. In a 4-GPU training setup, PPLL accelerated local learning training on ViT and ResNet by 162% and 33%, respectively, achieving 1.25x and 0.85x the speed of traditional pipeline parallelism.