A Note on Over-Smoothing for Graph Neural Networks

TL;DR

This paper analyzes the over-smoothing phenomenon in graph neural networks, showing how spectral properties of the weight matrix lead to loss of discriminative power as layers increase.

Contribution

It extends previous linear analyses to general GNNs, using Dirichlet energy to provide a clearer, more general understanding of over-smoothing effects.

Findings

Dirichlet energy converges to zero under certain spectral conditions

Over-smoothing causes embeddings to lose discriminative power

Analysis applies to non-linear GNN architectures

Abstract

Graph Neural Networks (GNNs) have achieved a lot of success on graph-structured data. However, it is observed that the performance of graph neural networks does not improve as the number of layers increases. This effect, known as over-smoothing, has been analyzed mostly in linear cases. In this paper, we build upon previous results \cite{oono2019graph} to further analyze the over-smoothing effect in the general graph neural network architecture. We show when the weight matrix satisfies the conditions determined by the spectrum of augmented normalized Laplacian, the Dirichlet energy of embeddings will converge to zero, resulting in the loss of discriminative power. Using Dirichlet energy to measure "expressiveness" of embedding is conceptually clean; it leads to simpler proofs than \cite{oono2019graph} and can handle more non-linearities.