Are Graph Transformers Necessary? Efficient Long-Range Message Passing with Fractal Nodes in MPNNs

TL;DR

This paper introduces fractal nodes in MPNNs to enhance long-range message passing, achieving comparable or better performance than graph Transformers with improved efficiency and addressing the over-squashing problem.

Contribution

The paper proposes fractal nodes that integrate subgraph features into MPNNs, improving long-range dependencies and computational efficiency over existing methods.

Findings

Improved expressive power of MPNNs with fractal nodes

Achieved comparable or better performance than graph Transformers

Enhanced long-range message passing efficiency

Abstract

Graph Neural Networks (GNNs) have emerged as powerful tools for learning on graph-structured data, but often struggle to balance local and global information. While graph Transformers aim to address this by enabling long-range interactions, they often overlook the inherent locality and efficiency of Message Passing Neural Networks (MPNNs). We propose a new concept called fractal nodes, inspired by the fractal structure observed in real-world networks. Our approach is based on the intuition that graph partitioning naturally induces fractal structure, where subgraphs often reflect the connectivity patterns of the full graph. Fractal nodes are designed to coexist with the original nodes and adaptively aggregate subgraph-level feature representations, thereby enforcing feature similarity within each subgraph. We show that fractal nodes alleviate the over-squashing problem by providing direct shortcut connections that enable long-range propagation of subgraph-level representations. Experiment results show that our method improves the expressive power of MPNNs and achieves comparable or better performance to graph Transformers while maintaining the computational efficiency of MPNN by improving the long-range dependencies of MPNN.