GSplit: Scaling Graph Neural Network Training on Large Graphs via Split-Parallelism

TL;DR

GSplit introduces split parallelism for GNN training on large graphs, reducing redundant computation and improving scalability by intelligently partitioning workloads across GPUs.

Contribution

The paper proposes a novel split parallelism paradigm and a lightweight partitioning algorithm to enhance GNN training efficiency on large graphs.

Findings

GSplit outperforms existing systems like DGL, Quiver, and P^3 in training large graphs.

Split parallelism reduces redundant work in GNN mini-batch training.

The partitioning algorithm minimizes communication overheads effectively.

Abstract

Graph neural networks (GNNs), an emerging class of machine learning models for graphs, have gained popularity for their superior performance in various graph analytical tasks. Mini-batch training is commonly used to train GNNs on large graphs, and data parallelism is the standard approach to scale mini-batch training across multiple GPUs. Data parallel approaches contain redundant work as subgraphs sampled by different GPUs contain significant overlap. To address this issue, we introduce a hybrid parallel mini-batch training paradigm called split parallelism. Split parallelism avoids redundant work by splitting the sampling, loading, and training of each mini-batch across multiple GPUs. Split parallelism, however, introduces communication overheads that can be more than the savings from removing redundant work. We further present a lightweight partitioning algorithm that probabilistically minimizes these overheads. We implement split parallelism in GSplit and show that it outperforms state-of-the-art mini-batch training systems like DGL, Quiver, and $P^3$.