GraphGPT: Generative Pre-trained Graph Eulerian Transformer

TL;DR

GraphGPT introduces a novel self-supervised pre-trained transformer model for graphs, converting graphs into sequences via Eulerian paths, achieving state-of-the-art results and scalable to 2 billion parameters.

Contribution

The paper presents GraphGPT, a new graph transformer leveraging Eulerian paths for sequence conversion, enabling scalable pre-training and improved performance on graph tasks.

Findings

Achieves state-of-the-art results on OGB datasets.

Scales to 2 billion parameters while maintaining performance.

Demonstrates strong results in molecular and protein graph tasks.

Abstract

We introduceGraphGPT, a novel self-supervised generative pre-trained model for graph learning based on the Graph Eulerian Transformer (GET). First, we propose GET, which combines a standard transformer encoder or decoder architecture with an innovative graph-to-sequence transformation method. This method converts graphs or sampled subgraphs into sequences of tokens representing nodes, edges, and attributes in a reversible manner using Eulerian paths. We pre-train GET using either of the two self-supervised tasks: next-token prediction (NTP) and scheduled masked-token prediction (SMTP). The pre-trained model is then fine-tuned for downstream tasks such as graph-, edge-, and node-level prediction. Despite its simplicity, GraphGPT achieves performance comparable to or surpassing state-of-the-art methods on multiple large-scale Open Graph Benchmark (OGB) datasets. It demonstrates exceptional results on the molecular property prediction dataset PCQM4Mv2 and the protein-protein interaction dataset ogbl-ppa. Notably, generative pre-training enables scaling GraphGPT to 2 billion parameters while maintaining performance gains - a breakthrough that overcomes the scalability limitations of traditional Graph Neural Networks (GNNs) and prior graph transformers (GTs). To advance research in graph foundation models and facilitate scientific discovery in chemistry, materials science, and related fields, we will release the source code (https://github.com/alibaba/graph-gpt) and pre-trained checkpoints.