Enhanced Graph Transformer with Serialized Graph Tokens

TL;DR

This paper introduces a serialized token paradigm for graph transformers, improving global signal encoding and achieving state-of-the-art results on graph-level tasks by better modeling interactions among multiple graph tokens.

Contribution

The paper proposes a novel serialized token approach with graph serialization and positional encoding, enhancing global signal capture in graph transformers.

Findings

Achieves state-of-the-art results on graph benchmarks.

Effective modeling of complex interactions among graph tokens.

Ablation studies confirm the proposed modules' effectiveness.

Abstract

Transformers have demonstrated success in graph learning, particularly for node-level tasks. However, existing methods encounter an information bottleneck when generating graph-level representations. The prevalent single token paradigm fails to fully leverage the inherent strength of self-attention in encoding token sequences, and degenerates into a weighted sum of node signals. To address this issue, we design a novel serialized token paradigm to encapsulate global signals more effectively. Specifically, a graph serialization method is proposed to aggregate node signals into serialized graph tokens, with positional encoding being automatically involved. Then, stacked self-attention layers are applied to encode this token sequence and capture its internal dependencies. Our method can yield more expressive graph representations by modeling complex interactions among multiple graph tokens. Experimental results show that our method achieves state-of-the-art results on several graph-level benchmarks. Ablation studies verify the effectiveness of the proposed modules.