Exploring Graph-Transformer Out-of-Distribution Generalization Abilities

TL;DR

This paper investigates the out-of-distribution generalization capabilities of graph-transformer models, demonstrating their superior robustness over traditional message-passing neural networks across various benchmarks and proposing a novel analysis method.

Contribution

It systematically evaluates graph-transformers under distribution shifts, compares them with MPNNs, and introduces a model-agnostic clustering analysis approach for better understanding generalization.

Findings

Graph-transformers outperform MPNNs in OOD settings on most benchmarks.

Hybrid GT-MPNN backbones show strong generalization without specialized algorithms.

The proposed clustering analysis provides insights into domain alignment and class separation.

Abstract

Deep learning on graphs has shown remarkable success across numerous applications, including social networks, bio-physics, traffic networks, and recommendation systems. Regardless of their successes, current methods frequently depend on the assumption that training and testing data share the same distribution, a condition rarely met in real-world scenarios. While graph-transformer (GT) backbones have recently outperformed traditional message-passing neural networks (MPNNs) in multiple in-distribution (ID) benchmarks, their effectiveness under distribution shifts remains largely unexplored. In this work, we address the challenge of out-of-distribution (OOD) generalization for graph neural networks, with a special focus on the impact of backbone architecture. We systematically evaluate GT and hybrid backbones in OOD settings and compare them to MPNNs. To do so, we adapt several leading domain generalization (DG) algorithms to work with GTs and assess their performance on a benchmark designed to test a variety of distribution shifts. Our results reveal that GT and hybrid GT-MPNN backbones demonstrate stronger generalization ability compared to MPNNs, even without specialized DG algorithms (on four out of six benchmarks). Additionally, we propose a novel post-training analysis approach that compares the clustering structure of the entire ID and OOD test datasets, specifically examining domain alignment and class separation. Highlighting its model-agnostic design, the method yielded valuable insights into both GT and MPNN backbones and appears well suited for broader DG applications beyond graph learning, offering a deeper perspective on generalization abilities that goes beyond standard accuracy metrics. Together, our findings highlight the promise of graph-transformers for robust, real-world graph learning and set a new direction for future research in OOD generalization.