Accelerating Mini-batch HGNN Training by Reducing CUDA Kernels

TL;DR

HiFuse significantly accelerates mini-batch HGNN training on CPU-GPU systems by reorganizing data, reducing CUDA kernel launches, and overlapping CPU-GPU tasks, achieving over twice the speed of previous methods.

Contribution

This paper introduces HiFuse, a novel system that improves GPU utilization and training speed for HGNNs by data restructuring and offloading graph construction to CPU with parallel processing.

Findings

Achieves 2.38x average speedup over state-of-the-art solutions.

Reduces CUDA kernel launches by merging vertex feature matrices.

Enhances GPU utilization through CPU-GPU overlapping and parallelization.

Abstract

Heterogeneous graph neural networks (HGNNs) are essential for capturing the structure and semantic information in heterogeneous graphs. However, existing GPU-based solutions, such as PyTorch Geometric, suffer from low GPU utilization due to numerous short-execution-time and memory-bound CUDA kernels during HGNN training. To address this issue, we introduce HiFuse, an enhancement for PyTorch Geometric designed to accelerate mini-batch HGNN training on CPU-GPU systems. From the data perspective, we reorganize and merge multiple smaller vertex feature matrices into larger ones, enabling a single kernel to process larger data chunks. This efficiently exploits data locality, reduces the kernel launch overhead, and improves overall GPU utilization. From the workflow perspective, we sophisticatedly offload the construction of semantic graphs from GPU to CPU to reduce the number of CUDA kernels. To meet the parallelism requirements on CPU and ensure seamless execution between CPU and GPU, we employ parallelization techniques including multi-threading and asynchronous pipeline. This allows different stages of the process to overlap, enhancing GPU utilization and reducing end-to-end execution latency, leading to a more efficient and balanced use of computational resources. Through extensive experiments, HiFuse demonstrates an average 2.38 times speedup compared to a state-of-the-art solution.