Transformer for Graphs: An Overview from Architecture Perspective

TL;DR

This paper systematically reviews various Graph Transformer models, analyzing their architectural designs and evaluating their effectiveness across multiple graph tasks through comprehensive experiments.

Contribution

It provides a detailed classification of Graph Transformer architectures and empirically assesses their performance, offering insights into their advantages and limitations.

Findings

Graph-specific modules improve Transformer performance on graph tasks.

Different architectural components show varied effectiveness across tasks.

Systematic comparison clarifies the benefits of each design choice.

Abstract

Recently, Transformer model, which has achieved great success in many artificial intelligence fields, has demonstrated its great potential in modeling graph-structured data. Till now, a great variety of Transformers has been proposed to adapt to the graph-structured data. However, a comprehensive literature review and systematical evaluation of these Transformer variants for graphs are still unavailable. It's imperative to sort out the existing Transformer models for graphs and systematically investigate their effectiveness on various graph tasks. In this survey, we provide a comprehensive review of various Graph Transformer models from the architectural design perspective. We first disassemble the existing models and conclude three typical ways to incorporate the graph information into the vanilla Transformer: 1) GNNs as Auxiliary Modules, 2) Improved Positional Embedding from Graphs, and 3) Improved Attention Matrix from Graphs. Furthermore, we implement the representative components in three groups and conduct a comprehensive comparison on various kinds of famous graph data benchmarks to investigate the real performance gain of each component. Our experiments confirm the benefits of current graph-specific modules on Transformer and reveal their advantages on different kinds of graph tasks.