Generalizing Graph Transformers Across Diverse Graphs and Tasks via Pre-training

TL;DR

This paper introduces PGT, a scalable transformer-based graph pre-training framework that generalizes across diverse graphs and tasks, demonstrating state-of-the-art results on large-scale datasets and real-world industrial data.

Contribution

The paper presents a novel pre-training strategy using a masked autoencoder architecture with a decoder for feature augmentation, enabling generalization to unseen nodes and graphs.

Findings

Achieves state-of-the-art performance on ogbn-papers100M dataset.

Successfully pre-trains on real-world graphs with over 540 million nodes.

Demonstrates effective generalization across static and dynamic downstream tasks.

Abstract

Graph pre-training has been concentrated on graph-level tasks involving small graphs (e.g., molecular graphs) or learning node representations on a fixed graph. Extending graph pre-trained models to web-scale graphs with billions of nodes in industrial scenarios, while avoiding negative transfer across graphs or tasks, remains a challenge. We aim to develop a general graph pre-trained model with inductive ability that can make predictions for unseen new nodes and even new graphs. In this work, we introduce a scalable transformer-based graph pre-training framework called PGT (Pre-trained Graph Transformer). Based on the masked autoencoder architecture, we design two pre-training tasks: one for reconstructing node features and the other for reconstructing local structures. Unlike the original autoencoder architecture where the pre-trained decoder is discarded, we propose a novel strategy that utilizes the decoder for feature augmentation. Our framework, tested on the publicly available ogbn-papers100M dataset with 111 million nodes and 1.6 billion edges, achieves state-of-the-art performance, showcasing scalability and efficiency. We have deployed our framework on Tencent's online game data, confirming its capability to pre-train on real-world graphs with over 540 million nodes and 12 billion edges and to generalize effectively across diverse static and dynamic downstream tasks.