Understanding and Resolving Performance Degradation in Graph Convolutional Networks

TL;DR

This paper investigates the causes of performance degradation in deep Graph Convolutional Networks (GCNs), highlighting the role of transformation operations and proposing a normalization technique to improve deep GCN performance.

Contribution

It reveals the significant impact of transformation operations on GCN performance decline and introduces NodeNorm, a simple normalization method to mitigate variance inflammation in deep GCNs.

Findings

NodeNorm improves deep GCN performance on benchmark datasets.

Transformation operations significantly contribute to performance degradation.

NodeNorm generalizes well to other GNN architectures.

Abstract

A Graph Convolutional Network (GCN) stacks several layers and in each layer performs a PROPagation operation (PROP) and a TRANsformation operation (TRAN) for learning node representations over graph-structured data. Though powerful, GCNs tend to suffer performance drop when the model gets deep. Previous works focus on PROPs to study and mitigate this issue, but the role of TRANs is barely investigated. In this work, we study performance degradation of GCNs by experimentally examining how stacking only TRANs or PROPs works. We find that TRANs contribute significantly, or even more than PROPs, to declining performance, and moreover that they tend to amplify node-wise feature variance in GCNs, causing variance inflammation that we identify as a key factor for causing performance drop. Motivated by such observations, we propose a variance-controlling technique termed Node Normalization (NodeNorm), which scales each node's features using its own standard deviation. Experimental results validate the effectiveness of NodeNorm on addressing performance degradation of GCNs. Specifically, it enables deep GCNs to outperform shallow ones in cases where deep models are needed, and to achieve comparable results with shallow ones on 6 benchmark datasets. NodeNorm is a generic plug-in and can well generalize to other GNN architectures. Code is publicly available at https://github.com/miafei/NodeNorm.