Exploring and Improving Initialization for Deep Graph Neural Networks: A Signal Propagation Perspective

TL;DR

This paper introduces SPoGInit, a new initialization method for deep GNNs based on signal propagation analysis, significantly improving performance and enabling deeper networks.

Contribution

The paper develops a signal propagation framework for GNN initialization and proposes SPoGInit, a method that optimizes three key metrics to enhance deep GNN training.

Findings

SPoGInit outperforms standard initialization methods.

Deep GNNs benefit from improved signal propagation.

The framework addresses depth-related performance degradation.

Abstract

Graph Neural Networks (GNNs) often suffer from performance degradation as the network depth increases. This paper addresses this issue by introducing initialization methods that enhance signal propagation (SP) within GNNs. We propose three key metrics for effective SP in GNNs: forward propagation, backward propagation, and graph embedding variation (GEV). While the first two metrics derive from classical SP theory, the third is specifically designed for GNNs. We theoretically demonstrate that a broad range of commonly used initialization methods for GNNs, which exhibit performance degradation with increasing depth, fail to control these three metrics simultaneously. To deal with this limitation, a direct exploitation of the SP analysis--searching for weight initialization variances that optimize the three metrics--is shown to significantly enhance the SP in deep GCNs. This approach is called Signal Propagation on Graph-guided Initialization (SPoGInit). Our experiments demonstrate that SPoGInit outperforms commonly used initialization methods on various tasks and architectures. Notably, SPoGInit enables performance improvements as GNNs deepen, which represents a significant advancement in addressing depth-related challenges and highlights the validity and effectiveness of the SP analysis framework.