PiPAD: Pipelined and Parallel Dynamic GNN Training on GPUs

TL;DR

PiPAD is a GPU training framework for dynamic GNNs that enhances performance by parallel processing multiple graph snapshots, reducing data transfer and memory access inefficiencies, achieving significant speedups.

Contribution

PiPAD introduces a pipelined and parallel training paradigm for DGNNs on GPUs, optimizing data organization and computation to improve efficiency.

Findings

Achieves up to 9.57x speedup over existing frameworks.

Effectively processes multiple graph snapshots in parallel.

Reduces data transfer and memory access overhead.

Abstract

Dynamic Graph Neural Networks (DGNNs) have been broadly applied in various real-life applications, such as link prediction and pandemic forecast, to capture both static structural information and temporal characteristics from dynamic graphs. Combining both time-dependent and -independent components, DGNNs manifest substantial parallel computation and data reuse potentials, but suffer from severe memory access inefficiency and data transfer overhead under the canonical one-graph-at-a-time training pattern. To tackle the challenges, we propose PiPAD, a $\underline{\textbf{Pi}}pelined$ and $\underline{\textbf{PA}}rallel$ $\underline{\textbf{D}}GNN$ training framework for the end-to-end performance optimization on GPUs. From both the algorithm and runtime level, PiPAD holistically reconstructs the overall training paradigm from the data organization to computation manner. Capable of processing multiple graph snapshots in parallel, PiPAD eliminates the unnecessary data transmission and alleviates memory access inefficiency to improve the overall performance. Our evaluation across various datasets shows PiPAD achieves $1.22\times$-$9.57\times$ speedup over the state-of-the-art DGNN frameworks on three representative models.