GraphFM: A Comprehensive Benchmark for Graph Foundation Model

TL;DR

This paper introduces GraphFM, a comprehensive benchmark for evaluating the generalization, scalability, and efficiency of Graph Foundation Models, providing insights to guide future research in graph self-supervised learning.

Contribution

It presents a rigorous benchmark for analyzing and comparing graph self-supervised models across multiple aspects including generalization, scalability, and training efficiency.

Findings

Self-supervised GNN models vary in generalization performance across tasks.

Scalability differs between full-batch and mini-batch training strategies.

Training efficiency metrics highlight the resource demands of different models.

Abstract

Foundation Models (FMs) serve as a general class for the development of artificial intelligence systems, offering broad potential for generalization across a spectrum of downstream tasks. Despite extensive research into self-supervised learning as the cornerstone of FMs, several outstanding issues persist in Graph Foundation Models that rely on graph self-supervised learning, namely: 1) Homogenization. The extent of generalization capability on downstream tasks remains unclear. 2) Scalability. It is unknown how effectively these models can scale to large datasets. 3) Efficiency. The training time and memory usage of these models require evaluation. 4) Training Stop Criteria. Determining the optimal stopping strategy for pre-training across multiple tasks to maximize performance on downstream tasks. To address these questions, we have constructed a rigorous benchmark that thoroughly analyzes and studies the generalization and scalability of self-supervised Graph Neural Network (GNN) models. Regarding generalization, we have implemented and compared the performance of various self-supervised GNN models, trained to generate node representations, across tasks such as node classification, link prediction, and node clustering. For scalability, we have compared the performance of various models after training using full-batch and mini-batch strategies. Additionally, we have assessed the training efficiency of these models by conducting experiments to test their GPU memory usage and throughput. Through these experiments, we aim to provide insights to motivate future research. The code for this benchmark is publicly available at https://github.com/NYUSHCS/GraphFM.