Neighbor-Sampling Based Momentum Stochastic Methods for Training Graph Neural Networks

TL;DR

This paper introduces neighbor-sampling based Adam-type stochastic methods with momentum for training GCNs, providing theoretical convergence guarantees and demonstrating superior performance on large-scale graph datasets.

Contribution

It develops novel neighbor-sampling Adam-type methods with momentum for GCN training, incorporating control variates for reduced variance and proven optimal convergence rates.

Findings

Superior performance over classic NS-based SGD

Effective on large-scale graph datasets

Theoretically optimal convergence rates achieved

Abstract

Graph convolutional networks (GCNs) are a powerful tool for graph representation learning. Due to the recursive neighborhood aggregations employed by GCNs, efficient training methods suffer from a lack of theoretical guarantees or are missing important practical elements from modern deep learning algorithms, such as adaptivity and momentum. In this paper, we present several neighbor-sampling (NS) based Adam-type stochastic methods for solving a nonconvex GCN training problem. We utilize the control variate technique proposed by [1] to reduce the stochastic error caused by neighbor sampling. Under standard assumptions for Adam-type methods, we show that our methods enjoy the optimal convergence rate. In addition, we conduct extensive numerical experiments on node classification tasks with several benchmark datasets. The results demonstrate superior performance of our methods over classic NS-based SGD that also uses the control-variate technique, especially for large-scale graph datasets. Our code is available at https://github.com/RPI-OPT/CV-ADAM-GNN .