FastGL: A GPU-Efficient Framework for Accelerating Sampling-Based GNN Training at Large Scale

TL;DR

FastGL is a GPU-efficient framework that accelerates large-scale sampling-based GNN training by optimizing sampling, memory IO, and computation, achieving significant speedups over existing frameworks.

Contribution

FastGL introduces a comprehensive GPU-optimized approach for sampling-based GNN training, including strategies to reduce data traffic, mitigate irregular data access, and decrease synchronization overhead.

Findings

Achieves an average speedup of 11.8x over PyG.

Reduces memory IO traffic with Match-Reorder strategy.

Diminishes synchronization overhead with Fused-Map approach.

Abstract

Graph Neural Networks (GNNs) have shown great superiority on non-Euclidean graph data, achieving ground-breaking performance on various graph-related tasks. As a practical solution to train GNN on large graphs with billions of nodes and edges, the sampling-based training is widely adopted by existing training frameworks. However, through an in-depth analysis, we observe that the efficiency of existing sampling-based training frameworks is still limited due to the key bottlenecks lying in all three phases of sampling-based training, i.e., subgraph sample, memory IO, and computation. To this end, we propose FastGL, a GPU-efficient Framework for accelerating sampling-based training of GNN at Large scale by simultaneously optimizing all above three phases, taking into account both GPU characteristics and graph structure. Specifically, by exploiting the inherent overlap within graph structures, FastGL develops the Match-Reorder strategy to reduce the data traffic, which accelerates the memory IO without incurring any GPU memory overhead. Additionally, FastGL leverages a Memory-Aware computation method, harnessing the GPU memory's hierarchical nature to mitigate irregular data access during computation. FastGL further incorporates the Fused-Map approach aimed at diminishing the synchronization overhead during sampling. Extensive experiments demonstrate that FastGL can achieve an average speedup of 11.8x, 2.2x and 1.5x over the state-of-the-art frameworks PyG, DGL, and GNNLab, respectively.Our code is available at https://github.com/a1bc2def6g/fastgl-ae.