Grappa: Gradient-Only Communication for Scalable Graph Neural Network Training

TL;DR

Grappa is a distributed GNN training framework that significantly reduces communication costs by exchanging only gradients, enabling faster training and better accuracy for large-scale graphs without high-bandwidth hardware.

Contribution

Grappa introduces gradient-only communication with unbiased gradient correction, allowing scalable, efficient GNN training at trillion-edge scale on commodity hardware.

Findings

Trains GNNs 4x faster on average compared to state-of-the-art systems.

Achieves better accuracy for deeper models.

Supports training at trillion-edge scale without high-bandwidth interconnects.

Abstract

Cross-partition edges dominate the cost of distributed GNN training: fetching remote features and activations per iteration overwhelms the network as graphs deepen and partition counts grow. Grappa is a distributed GNN training framework that enforces gradient-only communication: during each iteration, partitions train in isolation and exchange only gradients for the global update. To recover accuracy lost to isolation, Grappa (i) periodically repartitions to expose new neighborhoods and (ii) applies a lightweight coverage-corrected gradient aggregation inspired by importance sampling. We present an asymptotically unbiased estimator for gradient correction, which we use to develop a minimum-distance batch-level variant that is compatible with common deep-learning packages. We also introduce a shrinkage version that improves stability in practice. Empirical results on real and synthetic graphs show that Grappa trains GNNs 4x faster on average (up to 13x) than state-of-the-art systems, achieves better accuracy especially for deeper models, and sustains training at the trillion-edge scale on commodity hardware. Grappa is model-agnostic, supports full-graph and mini-batch training, and does not rely on high-bandwidth interconnects or caching.