Scalable Message Passing Neural Networks: No Need for Attention in Large Graph Representation Learning

TL;DR

This paper introduces SMPNNs, a scalable GNN architecture that replaces attention with convolutional message passing, achieving competitive performance on large graphs without the high computational costs of attention mechanisms.

Contribution

The paper presents a novel scalable GNN architecture that avoids attention, enabling deep message passing and improved performance on large graphs, supported by new theoretical insights.

Findings

SMPNNs outperform existing Graph Transformers in large graph tasks.

Residual connections are essential for preserving universality in graph convolutions.

The architecture scales to large graphs and allows deep message passing.

Abstract

We propose Scalable Message Passing Neural Networks (SMPNNs) and demonstrate that, by integrating standard convolutional message passing into a Pre-Layer Normalization Transformer-style block instead of attention, we can produce high-performing deep message-passing-based Graph Neural Networks (GNNs). This modification yields results competitive with the state-of-the-art in large graph transductive learning, particularly outperforming the best Graph Transformers in the literature, without requiring the otherwise computationally and memory-expensive attention mechanism. Our architecture not only scales to large graphs but also makes it possible to construct deep message-passing networks, unlike simple GNNs, which have traditionally been constrained to shallow architectures due to oversmoothing. Moreover, we provide a new theoretical analysis of oversmoothing based on universal approximation which we use to motivate SMPNNs. We show that in the context of graph convolutions, residual connections are necessary for maintaining the universal approximation properties of downstream learners and that removing them can lead to a loss of universality.