MQ-GNN: A Multi-Queue Pipelined Architecture for Scalable and Efficient GNN Training

TL;DR

MQ-GNN introduces a multi-queue pipelined architecture with asynchronous updates and caching to significantly improve the scalability and efficiency of multi-GPU GNN training, reducing training time and increasing GPU utilization.

Contribution

It proposes MQ-GNN, a novel framework that overlaps training stages, employs asynchronous gradient sharing, and optimizes resource use for scalable GNN training.

Findings

Achieves up to 4.6x faster training times

Improves GPU utilization by 30%

Maintains competitive accuracy across datasets

Abstract

Graph Neural Networks (GNNs) are powerful tools for learning graph-structured data, but their scalability is hindered by inefficient mini-batch generation, data transfer bottlenecks, and costly inter-GPU synchronization. Existing training frameworks fail to overlap these stages, leading to suboptimal resource utilization. This paper proposes MQ-GNN, a multi-queue pipelined framework that maximizes training efficiency by interleaving GNN training stages and optimizing resource utilization. MQ-GNN introduces Ready-to-Update Asynchronous Consistent Model (RaCoM), which enables asynchronous gradient sharing and model updates while ensuring global consistency through adaptive periodic synchronization. Additionally, it employs global neighbor sampling with caching to reduce data transfer overhead and an adaptive queue-sizing strategy to balance computation and memory efficiency. Experiments on four large-scale datasets and ten baseline models demonstrate that MQ-GNN achieves up to \boldmath $\bm{4.6\,\times}$ faster training time and 30% improved GPU utilization while maintaining competitive accuracy. These results establish MQ-GNN as a scalable and efficient solution for multi-GPU GNN training.