Solving Oversmoothing in GNNs via Nonlocal Message Passing: Algebraic Smoothing and Depth Scalability

TL;DR

This paper introduces a novel Post-LN based method for GNNs that induces algebraic smoothing, effectively preventing oversmoothing and enabling the training of significantly deeper networks without additional parameters.

Contribution

It provides a theoretical analysis of Layer Normalization effects and proposes a parameter-efficient solution to enhance GNN depth scalability and performance.

Findings

Supports GNNs with up to 256 layers

Improves performance across five benchmarks

Requires no additional parameters

Abstract

The relationship between Layer Normalization (LN) placement and the oversmoothing phenomenon remains underexplored. We identify a critical dilemma: Pre-LN architectures avoid oversmoothing but suffer from the curse of depth, while Post-LN architectures bypass the curse of depth but experience oversmoothing. To resolve this, we propose a new method based on Post-LN that induces algebraic smoothing, preventing oversmoothing without the curse of depth. Empirical results across five benchmarks demonstrate that our approach supports deeper networks (up to 256 layers) and improves performance, requiring no additional parameters. Key contributions: Theoretical Characterization: Analysis of LN dynamics and their impact on oversmoothing and the curse of depth. A Principled Solution: A parameter-efficient method that induces algebraic smoothing and avoids oversmoothing and the curse of depth. Empirical Validation: Extensive experiments showing the effectiveness of the method in deeper GNNs.