Communication-Efficient Sampling for Distributed Training of Graph Convolutional Networks

TL;DR

This paper introduces a communication-efficient neighbor sampling method for distributed GCN training, significantly reducing communication costs while maintaining accuracy.

Contribution

It proposes a novel sampling algorithm that prioritizes local nodes to minimize inter-machine communication in distributed GCN training.

Findings

Reduces communication overhead in distributed GCN training

Maintains comparable accuracy with traditional methods

Effective on node classification benchmarks

Abstract

Training Graph Convolutional Networks (GCNs) is expensive as it needs to aggregate data recursively from neighboring nodes. To reduce the computation overhead, previous works have proposed various neighbor sampling methods that estimate the aggregation result based on a small number of sampled neighbors. Although these methods have successfully accelerated the training, they mainly focus on the single-machine setting. As real-world graphs are large, training GCNs in distributed systems is desirable. However, we found that the existing neighbor sampling methods do not work well in a distributed setting. Specifically, a naive implementation may incur a huge amount of communication of feature vectors among different machines. To address this problem, we propose a communication-efficient neighbor sampling method in this work. Our main idea is to assign higher sampling probabilities to the local nodes so that remote nodes are accessed less frequently. We present an algorithm that determines the local sampling probabilities and makes sure our skewed neighbor sampling does not affect much the convergence of the training. Our experiments with node classification benchmarks show that our method significantly reduces the communication overhead for distributed GCN training with little accuracy loss.