GraphCLIP: Enhancing Transferability in Graph Foundation Models for Text-Attributed Graphs

TL;DR

GraphCLIP introduces a self-supervised contrastive pretraining framework for text-attributed graphs, significantly improving cross-domain zero-shot and few-shot transferability of graph foundation models using large-scale graph-summary data and prompt tuning.

Contribution

It presents a novel graph-summary pretraining method with invariant learning and a graph prompt tuning technique, enhancing transferability and reducing training costs.

Findings

Outperforms existing methods in zero-shot transfer tasks.

Demonstrates strong few-shot learning capabilities.

Versatile across various downstream graph tasks.

Abstract

Recently, research on Text-Attributed Graphs (TAGs) has gained significant attention due to the prevalence of free-text node features in real-world applications and the advancements in Large Language Models (LLMs) that bolster TAG methodologies. However, current TAG approaches face two primary challenges: (i) Heavy reliance on label information and (ii) Limited cross-domain zero/few-shot transferability. These issues constrain the scaling of both data and model size, owing to high labor costs and scaling laws, complicating the development of graph foundation models with strong transferability. In this work, we propose the GraphCLIP framework to address these challenges by learning graph foundation models with strong cross-domain zero/few-shot transferability through a self-supervised contrastive graph-summary pretraining method. Specifically, we generate and curate large-scale graph-summary pair data with the assistance of LLMs, and introduce a novel graph-summary pretraining method, combined with invariant learning, to enhance graph foundation models with strong cross-domain zero-shot transferability. For few-shot learning, we propose a novel graph prompt tuning technique aligned with our pretraining objective to mitigate catastrophic forgetting and minimize learning costs. Extensive experiments show the superiority of GraphCLIP in both zero-shot and few-shot settings, while evaluations across various downstream tasks confirm the versatility of GraphCLIP. Our code is available at: https://github.com/ZhuYun97/GraphCLIP