Deep Neural Sheaf Diffusion

TL;DR

This paper introduces Deep Neural Sheaf Diffusion (DNSD), a novel deep graph neural network method that overcomes representation collapse in deep layers by replacing the sheaf Laplacian with a sheaf adjacency operator, enabling effective deep aggregation.

Contribution

The paper proposes DNSD, a new deep GNN architecture that maintains informative signals across layers by modifying the sheaf diffusion process and incorporating normalization, nonlinearities, and gating.

Findings

DNSD outperforms GNN and NSD baselines with up to 30pp accuracy on synthetic datasets.

DNSD consistently outperforms baselines on real-world benchmarks.

Theoretical analysis contrasts sheaf diffusion with graph attention mechanisms.

Abstract

Deep Graph Neural Networks (GNNs) are essential for capturing complex dependencies in graph-structured data. However, scaling GNNs to depth remains challenging, as stacking layers leads to representation collapse and diminishing sensitivity due to repeated aggregation. While Neural Sheaf Diffusion (NSD) provides strong theoretical guarantees against such collapse, these guarantees do not translate to practice: as depth increases, the disagreement signal of the sheaf Laplacian vanishes, limiting the contribution of deeper layers. We identify mechanisms that hinder NSD effectiveness at depth and propose \emph{Deep Neural Sheaf Diffusion} (DNSD), which replaces the sheaf Laplacian with a sheaf adjacency operator to maintain informative signals across layers. This is complemented by normalization, odd nonlinearities, and gating. To provide a principled explanation of the expected performance improvement, we contrast sheaf diffusion to graph attention mechanisms, highlighting that DNSD replaces scalar attention scores with matrix-valued edge functions and normalizes node representations rather than attention scores. We demonstrate empirically that DNSD effectively utilizes deep aggregation in graph tasks, outperforming GNN and NSD baselines with up to 30pp accuracy on synthetic long-range datasets, and consistently outperforming them on real-world benchmarks. These results position sheaf-based architectures as a promising building block for graph foundation models by supporting effective deep architectures.