Rank Collapse Causes Over-Smoothing and Over-Correlation in Graph Neural Networks

TL;DR

This paper provides new theoretical insights into over-smoothing and feature over-correlation in graph neural networks, highlighting rank collapse as a key issue and proposing a Kronecker product approach to prevent it.

Contribution

It introduces a theoretical framework linking over-smoothing to rank collapse and proposes a novel Kronecker product method to mitigate these issues in GNNs.

Findings

Rank of node representations collapses with depth, causing over-smoothing.

Existing models struggle to fit target functions in node classification.

Proposed Kronecker product approach prevents over-smoothing and rank collapse.

Abstract

Our study reveals new theoretical insights into over-smoothing and feature over-correlation in graph neural networks. Specifically, we demonstrate that with increased depth, node representations become dominated by a low-dimensional subspace that depends on the aggregation function but not on the feature transformations. For all aggregation functions, the rank of the node representations collapses, resulting in over-smoothing for particular aggregation functions. Our study emphasizes the importance for future research to focus on rank collapse rather than over-smoothing. Guided by our theory, we propose a sum of Kronecker products as a beneficial property that provably prevents over-smoothing, over-correlation, and rank collapse. We empirically demonstrate the shortcomings of existing models in fitting target functions of node classification tasks.