Evaluating Progress in Graph Foundation Models: A Comprehensive Benchmark and New Insights

TL;DR

This paper introduces a comprehensive benchmark for graph foundation models that evaluates their ability to transfer knowledge across both topic and format domains, revealing new insights into their generalization capabilities.

Contribution

It presents a novel two-axis benchmark that jointly assesses topic and format domain shifts in GFMs, enabling more thorough evaluation of their transfer learning performance.

Findings

Extensive evaluation of 8 GFMs on 33 datasets.

Identifies key factors influencing transferability.

Provides practical insights for future GFM development.

Abstract

Graph foundation models (GFM) aim to acquire transferable knowledge by pre-training on diverse graphs, which can be adapted to various downstream tasks. However, domain shift in graphs is inherently two-dimensional: graphs differ not only in what they describe (topic domains) but also in how they are represented (format domains). Most existing GFM benchmarks vary only topic domains, thereby obscuring how knowledge transfers across both dimensions. We present a new benchmark that jointly evaluates topic and format gaps across the full GFM pipeline, including multi-domain self-supervised pre-training and few-shot downstream adaptation, and provides a timely evaluation of recent GFMs in the rapidly evolving landscape. Our protocol enables controlled assessment in four settings: (i) pre-training on diverse topics and formats, while adapting to unseen downstream datasets; (ii) same pre-training as in (i), while adapting to seen datasets; (iii) pre-training on a single topic domain, while adapting to other topics; (iv) pre-training on a base format, while adapting to other formats. This two-axis evaluation disentangles semantic generalization from robustness to representational shifts. We conduct extensive evaluations of eight state-of-the-art GFMs on 33 datasets spanning seven topic domains and six format domains, surfacing new empirical observations and practical insights for future research. Codes/data are available at https://github.com/smufang/GFMBenchmark.