GraphNorm: A Principled Approach to Accelerating Graph Neural Network Training

TL;DR

This paper introduces GraphNorm, a normalization method for GNNs that accelerates training and improves generalization by addressing limitations of existing normalization techniques like BatchNorm and InstanceNorm.

Contribution

The paper proposes GraphNorm with a learnable shift, providing a principled normalization approach that enhances GNN training speed and performance over existing methods.

Findings

GraphNorm converges faster than other normalization methods.

GraphNorm improves GNN generalization on benchmark datasets.

InstanceNorm acts as a preconditioner for GNNs, but with limitations.

Abstract

Normalization is known to help the optimization of deep neural networks. Curiously, different architectures require specialized normalization methods. In this paper, we study what normalization is effective for Graph Neural Networks (GNNs). First, we adapt and evaluate the existing methods from other domains to GNNs. Faster convergence is achieved with InstanceNorm compared to BatchNorm and LayerNorm. We provide an explanation by showing that InstanceNorm serves as a preconditioner for GNNs, but such preconditioning effect is weaker with BatchNorm due to the heavy batch noise in graph datasets. Second, we show that the shift operation in InstanceNorm results in an expressiveness degradation of GNNs for highly regular graphs. We address this issue by proposing GraphNorm with a learnable shift. Empirically, GNNs with GraphNorm converge faster compared to GNNs using other normalization. GraphNorm also improves the generalization of GNNs, achieving better performance on graph classification benchmarks.