Rethinking Federated Graph Foundation Models: A Graph-Language Alignment-based Approach

TL;DR

This paper introduces FedGALA, a novel federated graph foundation model framework that aligns language and graph models in a continuous space, improving knowledge transfer and task performance in distributed, privacy-sensitive environments.

Contribution

FedGALA uniquely employs contrastive learning for semantic-structural alignment and prompt tuning for efficient adaptation, addressing knowledge loss and communication challenges in federated graph models.

Findings

Outperforms baselines with up to 14.37% performance gain

Effectively aligns GNNs and PLMs in a continuous embedding space

Reduces communication overhead via prompt tuning

Abstract

Recent studies of federated graph foundational models (FedGFMs) break the idealized and untenable assumption of having centralized data storage to train graph foundation models, and accommodate the reality of distributed, privacy-restricted data silos. Despite their simplicity and intuition, existing studies that project aligned generalizable knowledge onto a discrete token space via vector-quantized backbones suffer from irreversible knowledge loss during the quantization process. In this context, we argue that reconciling the semantic-structural orthogonality and integrity between pre-trained language models (PLMs) and graph neural networks (GNNs) is paramount for developing effective FedGFMs while simultaneously mitigating the severe data heterogeneity and communication constraints inherent in distributed, resource-limited environments. To address these issues, we propose FedGALA (Federated Graph And Language Alignment), a framework that resolves graph-based semantic-structural orthogonality and integrity in federated settings by employing unsupervised contrastive learning to align GNNs and frozen PLMs within a continuous embedding space, thereby capturing robust, transferable general knowledge. Subsequently, FedGALA leverages a communication-efficient prompt tuning mechanism to steer these pre-aligned encoders and frozen PLMs, facilitating effective adaptation to diverse downstream tasks while circumventing the prohibitive overhead of full-parameter fine-tuning. The comprehensive experiments validate that FedGALA outperforms all competitive baselines across multi-domain datasets on multiple tasks with up to 14.37% performance improvement.