Gaussian Process Limit Reveals Structural Benefits of Graph Transformers

TL;DR

This paper provides a theoretical analysis of graph transformers, showing they have structural advantages over traditional graph convolutional networks, especially in preserving community information and preventing oversmoothing, supported by empirical validation.

Contribution

It introduces a Gaussian process framework for analyzing graph transformers, revealing their ability to maintain discriminative features and community structure in deep layers, which is a novel theoretical insight.

Findings

Graph transformers structurally preserve community information.

They maintain discriminative node representations in deep layers.

Empirical results validate theoretical predictions and show performance improvements.

Abstract

Graph transformers are the state-of-the-art for learning from graph-structured data and are empirically known to avoid several pitfalls of message-passing architectures. However, there is limited theoretical analysis on why these models perform well in practice. In this work, we prove that attention-based architectures have structural benefits over graph convolutional networks in the context of node-level prediction tasks. Specifically, we study the neural network gaussian process limits of graph transformers (GAT, Graphormer, Specformer) with infinite width and infinite heads, and derive the node-level and edge-level kernels across the layers. Our results characterise how the node features and the graph structure propagate through the graph attention layers. As a specific example, we prove that graph transformers structurally preserve community information and maintain discriminative node representations even in deep layers, thereby preventing oversmoothing. We provide empirical evidence on synthetic and real-world graphs that validate our theoretical insights, such as integrating informative priors and positional encoding can improve performance of deep graph transformers.