Towards Foundation Models on Graphs: An Analysis on Cross-Dataset Transfer of Pretrained GNNs

TL;DR

This paper investigates the transferability of pretrained Graph Neural Networks across datasets, exploring how structural and feature information affect their generalization in downstream tasks.

Contribution

It introduces an extension to capture feature information in pretrained GNNs while maintaining dataset-agnosticism, and evaluates transferability across various datasets and training sizes.

Findings

Pretrained embeddings improve generalization with sufficient downstream data.

Feature information enhances transferability when pretraining and downstream features are similar.

Transfer effectiveness depends on the quantity and properties of pretraining data.

Abstract

To develop a preliminary understanding towards Graph Foundation Models, we study the extent to which pretrained Graph Neural Networks can be applied across datasets, an effort requiring to be agnostic to dataset-specific features and their encodings. We build upon a purely structural pretraining approach and propose an extension to capture feature information while still being feature-agnostic. We evaluate pretrained models on downstream tasks for varying amounts of training samples and choices of pretraining datasets. Our preliminary results indicate that embeddings from pretrained models improve generalization only with enough downstream data points and in a degree which depends on the quantity and properties of pretraining data. Feature information can lead to improvements, but currently requires some similarities between pretraining and downstream feature spaces.