MixGCN: Scalable GCN Training by Mixture of Parallelism and Mixture of Accelerators

TL;DR

MixGCN introduces a novel training framework for GCNs that combines multiple parallelism strategies and heterogeneous accelerators, significantly improving scalability and efficiency in full-graph training.

Contribution

It proposes a combined parallelism and accelerator approach to address memory and computation challenges in scalable GCN training.

Findings

Achieves constant communication volume with mixture of parallelism.

Enhances workload balance through theoretical and empirical analysis.

Demonstrates improved training efficiency and scalability in experiments.

Abstract

Graph convolutional networks (GCNs) have demonstrated superiority in graph-based learning tasks. However, training GCNs on full graphs is particularly challenging, due to the following two challenges: (1) the associated feature tensors can easily explode the memory and block the communication bandwidth of modern accelerators, and (2) the computation workflow in training GCNs alternates between sparse and dense matrix operations, complicating the efficient utilization of computational resources. Existing solutions for scalable distributed full-graph GCN training mostly adopt partition parallelism, which is unsatisfactory as they only partially address the first challenge while incurring scaled-out communication volume. To this end, we propose MixGCN aiming to simultaneously address both the aforementioned challenges towards GCN training. To tackle the first challenge, MixGCN integrates mixture of parallelism. Both theoretical and empirical analysis verify its constant communication volumes and enhanced balanced workload; For handling the second challenge, we consider mixture of accelerators (i.e., sparse and dense accelerators) with a dedicated accelerator for GCN training and a fine-grain pipeline. Extensive experiments show that MixGCN achieves boosted training efficiency and scalability.