DeeperGCN: All You Need to Train Deeper GCNs

TL;DR

DeeperGCN introduces new techniques to enable the training of very deep Graph Convolutional Networks, overcoming traditional issues like vanishing gradients and over-smoothing, and demonstrates improved performance on large-scale graph tasks.

Contribution

The paper proposes differentiable generalized aggregation functions, a novel MsgNorm layer, and pre-activation residual connections to facilitate training deeper GCNs.

Findings

DeeperGCN outperforms state-of-the-art models on large-scale graph benchmarks.

The new methods enable stable training of very deep GCN architectures.

Significant performance improvements are observed in node and graph property prediction tasks.

Abstract

Graph Convolutional Networks (GCNs) have been drawing significant attention with the power of representation learning on graphs. Unlike Convolutional Neural Networks (CNNs), which are able to take advantage of stacking very deep layers, GCNs suffer from vanishing gradient, over-smoothing and over-fitting issues when going deeper. These challenges limit the representation power of GCNs on large-scale graphs. This paper proposes DeeperGCN that is capable of successfully and reliably training very deep GCNs. We define differentiable generalized aggregation functions to unify different message aggregation operations (e.g. mean, max). We also propose a novel normalization layer namely MsgNorm and a pre-activation version of residual connections for GCNs. Extensive experiments on Open Graph Benchmark (OGB) show DeeperGCN significantly boosts performance over the state-of-the-art on the large scale graph learning tasks of node property prediction and graph property prediction. Please visit https://www.deepgcns.org for more information.