How Wide and How Deep? Mitigating Over-Squashing of GNNs via Channel Capacity Constrained Estimation

TL;DR

This paper introduces C3E, a framework based on information theory that optimally selects hidden dimensions and propagation depth in GNNs to mitigate over-squashing and enhance representation learning.

Contribution

It formulates hidden dimension and depth selection as a nonlinear programming problem rooted in spectral graph neural network channel capacity modeling.

Findings

C3E effectively mitigates over-squashing across nine datasets.

Increasing hidden dimensions reduces information compression in GNNs.

Propagation depth has a nuanced role in information retention.

Abstract

Existing graph neural networks typically rely on heuristic choices for hidden dimensions and propagation depths, which often lead to severe information loss during propagation, known as over-squashing. To address this issue, we propose Channel Capacity Constrained Estimation (C3E), a novel framework that formulates the selection of hidden dimensions and depth as a nonlinear programming problem grounded in information theory. Through modeling spectral graph neural networks as communication channels, our approach directly connects channel capacity to hidden dimensions, propagation depth, propagation mechanism, and graph structure. Extensive experiments on nine public datasets demonstrate that hidden dimensions and depths estimated by C3E can mitigate over-squashing and consistently improve representation learning. Experimental results show that over-squashing occurs due to the cumulative compression of information in representation matrices. Furthermore, our findings show that increasing hidden dimensions indeed mitigate information compression, while the role of propagation depth is more nuanced, uncovering a fundamental balance between information compression and representation complexity.