H$^2$GFM: Towards unifying Homogeneity and Heterogeneity on Text-Attributed Graphs

TL;DR

H$^2$GFM introduces a unified framework that effectively models both homogeneous and heterogeneous text-attributed graphs, leveraging a context-adaptive transformer and mixture of experts to improve generalization across diverse graph types.

Contribution

The paper proposes H$^2$GFM, a novel model that unifies handling of homogeneous and heterogeneous text-attributed graphs using a context encoding and mixture of graph transformer experts.

Findings

Effective across diverse graph types and tasks

Outperforms existing models on multiple benchmarks

Captures complex structural and semantic relationships

Abstract

The growing interests and applications of graph learning in diverse domains have propelled the development of a unified model generalizing well across different graphs and tasks, known as the Graph Foundation Model (GFM). Existing research has leveraged text-attributed graphs (TAGs) to tackle the heterogeneity in node features among graphs. However, they primarily focus on homogeneous TAGs (HoTAGs), leaving heterogeneous TAGs (HeTAGs), where multiple types of nodes/edges reside, underexplored. To enhance the capabilities and applications of GFM, we introduce H$^2$GFM, a novel framework designed to generalize across both HoTAGs and HeTAGs. Our model projects diverse meta-relations among graphs under a unified textual space, and employs a context encoding to capture spatial and higher-order semantic relationships. To achieve robust node representations, we propose a novel context-adaptive graph transformer (CGT), effectively capturing information from both context neighbors and their relationships. Furthermore, we employ a mixture of CGT experts to capture the heterogeneity in structural patterns among graph types. Comprehensive experiments on a wide range of HoTAGs and HeTAGs as well as learning scenarios demonstrate the effectiveness of our model.