Communication-free Sampling and 4D Hybrid Parallelism for Scalable Mini-batch GNN Training

TL;DR

ScaleGNN introduces a communication-free sampling method and 4D parallelism to significantly improve the scalability and efficiency of mini-batch GNN training on large GPU clusters.

Contribution

The paper proposes a novel 4D parallel framework combining communication-free sampling and 3D matrix multiplication for scalable GNN training.

Findings

Achieves 3.5x speedup over SOTA on ogbn-products.

Scales to 2048 GPUs with strong performance.

Reduces communication overhead significantly.

Abstract

Graph neural networks (GNNs) are widely used for learning on graph datasets derived from various real-world scenarios. Learning from extremely large graphs requires distributed training, and mini-batching with sampling is a popular approach for parallelizing GNN training. Existing distributed mini-batch approaches have significant performance bottlenecks due to expensive sampling methods and limited scaling when using data parallelism. In this work, we present ScaleGNN, a 4D parallel framework for scalable mini-batch GNN training that combines communication-free distributed sampling, 3D parallel matrix multiplication (PMM), and data parallelism. ScaleGNN introduces a uniform vertex sampling algorithm, enabling each process (GPU device) to construct its local mini-batch, i.e., subgraph partitions without any inter-process communication. 3D PMM enables scaling mini-batch training to much larger GPU counts than vanilla data parallelism with significantly lower communication overheads. We also present additional optimizations to overlap sampling with training, reduce communication overhead by sending data in lower precision, kernel fusion, and communication-computation overlap. We evaluate ScaleGNN on five graph datasets and demonstrate strong scaling up to 2048 GPUs on Perlmutter, 2048 GCDs on Frontier, and 1024 GPUs on Tuolumne. On Perlmutter, ScaleGNN achieves 3.5x end-to-end training speedup over the SOTA baseline on ogbn-products.