A Survey of Graph Transformers: Architectures, Theories and Applications

TL;DR

This survey comprehensively reviews Graph Transformers, detailing their architectures, theoretical foundations, and diverse applications, highlighting recent advancements and future research directions in this rapidly evolving field.

Contribution

It provides a systematic categorization of Graph Transformer architectures, analyzes their expressivity, and summarizes their practical applications across multiple domains.

Findings

Graph Transformers improve over GNNs in modeling complex graph structures.

Various architectures utilize graph tokenization, positional encoding, and structure-aware attention.

Graph Transformers are applied successfully in molecular, biological, language, vision, and traffic data analysis.

Abstract

Graph Transformers (GTs) have demonstrated a strong capability in modeling graph structures by addressing the intrinsic limitations of graph neural networks (GNNs), such as over-smoothing and over-squashing. Recent studies have proposed diverse architectures, enhanced explainability, and practical applications for Graph Transformers. In light of these rapid developments, we conduct a comprehensive review of Graph Transformers, covering aspects such as their architectures, theoretical foundations, and applications within this survey. We categorize the architecture of Graph Transformers according to their strategies for processing structural information, including graph tokenization, positional encoding, structure-aware attention and model ensemble. Furthermore, from the theoretical perspective, we examine the expressivity of Graph Transformers in various discussed architectures and contrast them with other advanced graph learning algorithms to discover the connections. Furthermore, we provide a summary of the practical applications where Graph Transformers have been utilized, such as molecule, protein, language, vision, traffic, brain and material data. At the end of this survey, we will discuss the current challenges and prospective directions in Graph Transformers for potential future research.