Distributed Training of Graph Convolutional Networks using Subgraph Approximation

TL;DR

This paper introduces a distributed training method for Graph Convolutional Networks that uses subgraph approximation to maintain accuracy across partitions while reducing memory use and synchronization costs.

Contribution

It proposes a novel subgraph approximation scheme that preserves information across graph partitions, enabling accurate distributed GCN training.

Findings

Achieves single-machine accuracy in distributed training

Reduces memory footprint significantly

Minimizes synchronization overhead

Abstract

Modern machine learning techniques are successfully being adapted to data modeled as graphs. However, many real-world graphs are typically very large and do not fit in memory, often making the problem of training machine learning models on them intractable. Distributed training has been successfully employed to alleviate memory problems and speed up training in machine learning domains in which the input data is assumed to be independently identical distributed (i.i.d). However, distributing the training of non i.i.d data such as graphs that are used as training inputs in Graph Convolutional Networks (GCNs) causes accuracy problems since information is lost at the graph partitioning boundaries. In this paper, we propose a training strategy that mitigates the lost information across multiple partitions of a graph through a subgraph approximation scheme. Our proposed approach augments each sub-graph with a small amount of edge and vertex information that is approximated from all other sub-graphs. The subgraph approximation approach helps the distributed training system converge at single-machine accuracy, while keeping the memory footprint low and minimizing synchronization overhead between the machines.