GMoPE:A Prompt-Expert Mixture Framework for Graph Foundation Models

TL;DR

GMoPE introduces a prompt-expert mixture framework for graph models that enhances domain generalization and efficiency through expert-specific prompts, structure-aware routing, and orthogonality constraints, outperforming existing methods.

Contribution

The paper proposes GMoPE, a novel graph foundation model framework combining Mixture-of-Experts with prompt-based learning, enabling scalable, diverse, and transferable graph representations.

Findings

GMoPE outperforms state-of-the-art baselines across multiple tasks.

It achieves comparable performance to full fine-tuning with less adaptation overhead.

The framework promotes expert diversity and specialization through orthogonality constraints.

Abstract

Graph Neural Networks (GNNs) have demonstrated impressive performance on task-specific benchmarks, yet their ability to generalize across diverse domains and tasks remains limited. Existing approaches often struggle with negative transfer, scalability issues, and high adaptation costs. To address these challenges, we propose GMoPE (Graph Mixture of Prompt-Experts), a novel framework that seamlessly integrates the Mixture-of-Experts (MoE) architecture with prompt-based learning for graphs. GMoPE leverages expert-specific prompt vectors and structure-aware MoE routing to enable each expert to specialize in distinct subdomains and dynamically contribute to predictions. To promote diversity and prevent expert collapse, we introduce a soft orthogonality constraint across prompt vectors, encouraging expert specialization and facilitating a more balanced expert utilization. Additionally, we adopt a prompt-only fine-tuning strategy that significantly reduces spatiotemporal complexity during transfer. We validate GMoPE through extensive experiments under various pretraining strategies and multiple downstream tasks. Results show that GMoPE consistently outperforms state-of-the-art baselines and achieves performance comparable to full parameter fine-tuning-while requiring only a fraction of the adaptation overhead. Our work provides a principled and scalable framework for advancing generalizable and efficient graph foundation models.