Size Transferability of Graph Transformers with Convolutional Positional Encodings

TL;DR

This paper investigates the transferability of Graph Transformers with convolutional positional encodings, providing theoretical guarantees for their generalization from small to large graphs and demonstrating their scalability and efficiency in practical tasks.

Contribution

It establishes a theoretical connection between Graph Transformers with GNN-based positional encodings and Manifold Neural Networks, proving their transferability and scalability.

Findings

GTs with GNN positional encodings inherit transferability guarantees.

GTs trained on small graphs generalize to larger graphs.

Experimental results show GTs are scalable and efficient in real-world tasks.

Abstract

Transformers have achieved remarkable success across domains, motivating the rise of Graph Transformers (GTs) as attention-based architectures for graph-structured data. A key design choice in GTs is the use of Graph Neural Network (GNN)-based positional encodings to incorporate structural information. In this work, we study GTs through the lens of manifold limit models for graph sequences and establish a theoretical connection between GTs with GNN positional encodings and Manifold Neural Networks (MNNs). Building on transferability results for GNNs under manifold convergence, we show that GTs inherit transferability guarantees from their positional encodings. In particular, GTs trained on small graphs provably generalize to larger graphs under mild assumptions. We complement our theory with extensive experiments on standard graph benchmarks, demonstrating that GTs exhibit scalable behavior on par with GNNs. To further show the efficiency in a real-world scenario, we implement GTs for shortest path distance estimation over terrains to better illustrate the efficiency of the transferable GTs. Our results provide new insights into the understanding of GTs and suggest practical directions for efficient training of GTs in large-scale settings.