Distributed Graph Neural Network Training with Periodic Stale Representation Synchronization

TL;DR

This paper introduces DIGEST, a distributed GNN training framework that combines the strengths of existing methods by using stale neighbor representations, achieving faster training with maintained accuracy on large graphs.

Contribution

DIGEST is a novel framework that leverages stale neighbor representations to balance communication efficiency and information preservation in distributed GNN training.

Findings

Achieves up to 21.82x speedup over existing methods.

Maintains comparable performance while reducing communication costs.

Provides theoretical convergence guarantees.

Abstract

Despite the recent success of Graph Neural Networks, it remains challenging to train a GNN on large graphs with millions of nodes and billions of edges, which are prevalent in many graph-based applications. Traditional sampling-based methods accelerate GNN training by dropping edges and nodes, which impairs the graph integrity and model performance. Differently, distributed GNN algorithms accelerate GNN training by utilizing multiple computing devices and can be classified into two types: "partition-based" methods enjoy low communication costs but suffer from information loss due to dropped edges, while "propagation-based" methods avoid information loss but suffer from prohibitive communication overhead caused by the neighbor explosion. To jointly address these problems, this paper proposes DIGEST (DIstributed Graph reprEsentation SynchronizaTion), a novel distributed GNN training framework that synergizes the complementary strength of both categories of existing methods. We propose to allow each device to utilize the stale representations of its neighbors in other subgraphs during subgraph parallel training. This way, our method preserves global graph information from neighbors to avoid information loss and reduce communication costs. Our convergence analysis demonstrates that DIGEST enjoys a state-of-the-art convergence rate. Extensive experimental evaluation on large, real-world graph datasets shows that DIGEST achieves up to 21.82 speedups without compromising performance compared to state-of-the-art distributed GNN training frameworks.