GPT-GNN: Generative Pre-Training of Graph Neural Networks

TL;DR

GPT-GNN introduces a generative pre-training framework for GNNs using self-supervised attributed graph generation, significantly improving downstream task performance on large-scale datasets by capturing structural and semantic properties.

Contribution

The paper proposes GPT-GNN, a novel self-supervised pre-training method for GNNs that models attribute and edge generation to enhance downstream task performance.

Findings

Outperforms state-of-the-art GNNs by up to 9.1% on large-scale datasets.

Effectively captures structural and semantic graph properties during pre-training.

Demonstrates strong transferability to various downstream tasks.

Abstract

Graph neural networks (GNNs) have been demonstrated to be powerful in modeling graph-structured data. However, training GNNs usually requires abundant task-specific labeled data, which is often arduously expensive to obtain. One effective way to reduce the labeling effort is to pre-train an expressive GNN model on unlabeled data with self-supervision and then transfer the learned model to downstream tasks with only a few labels. In this paper, we present the GPT-GNN framework to initialize GNNs by generative pre-training. GPT-GNN introduces a self-supervised attributed graph generation task to pre-train a GNN so that it can capture the structural and semantic properties of the graph. We factorize the likelihood of the graph generation into two components: 1) Attribute Generation and 2) Edge Generation. By modeling both components, GPT-GNN captures the inherent dependency between node attributes and graph structure during the generative process. Comprehensive experiments on the billion-scale Open Academic Graph and Amazon recommendation data demonstrate that GPT-GNN significantly outperforms state-of-the-art GNN models without pre-training by up to 9.1% across various downstream tasks.