Universal Graph Transformer Self-Attention Networks

TL;DR

This paper introduces UGformer, a transformer-based GNN model with two variants that achieve state-of-the-art accuracy on graph and text classification benchmarks, advancing graph neural network capabilities.

Contribution

The paper presents two novel UGformer variants leveraging transformers for graph learning, with one variant also applied successfully to text classification.

Findings

First UGformer variant achieves state-of-the-art on graph classification datasets.

Second UGformer variant achieves state-of-the-art on inductive text classification.

Both variants outperform previous models in their respective tasks.

Abstract

We introduce a transformer-based GNN model, named UGformer, to learn graph representations. In particular, we present two UGformer variants, wherein the first variant (publicized in September 2019) is to leverage the transformer on a set of sampled neighbors for each input node, while the second (publicized in May 2021) is to leverage the transformer on all input nodes. Experimental results demonstrate that the first UGformer variant achieves state-of-the-art accuracies on benchmark datasets for graph classification in both inductive setting and unsupervised transductive setting; and the second UGformer variant obtains state-of-the-art accuracies for inductive text classification. The code is available at: \url{https://github.com/daiquocnguyen/Graph-Transformer}.