AnyGraph: Graph Foundation Model in the Wild
Lianghao Xia, Chao Huang
TL;DR
AnyGraph is a versatile graph foundation model that addresses heterogeneity, enables fast domain adaptation, and exhibits favorable scaling behavior, demonstrated through extensive experiments on diverse datasets.
Contribution
The paper introduces AnyGraph, a unified graph model using MoE architecture that handles heterogeneity, enables rapid adaptation, and exhibits scaling laws, advancing graph foundation modeling.
Findings
Strong zero-shot performance across 38 diverse graph datasets.
Effective handling of structure and feature heterogeneity.
Demonstrated fast adaptation and scaling law emergence.
Abstract
The growing ubiquity of relational data structured as graphs has underscored the need for graph learning models with exceptional generalization capabilities. However, current approaches often struggle to effectively extract generalizable insights, frequently requiring extensive fine-tuning and limiting their versatility. Graph foundation models offer a transformative solution, with the potential to learn robust, generalizable representations from graph data. This enables more effective and adaptable applications across a wide spectrum of tasks and domains. In this work, we investigate a unified graph model, AnyGraph, designed to handle key challenges: i) Structure Heterogenity. Addressing distribution shift in graph structural information; ii) Feature Heterogenity. Handling diverse feature representation spaces across graph datasets; iii) Fast Adaptation. Efficiently adapting the model…
Peer Reviews
Decision·ICLR 2025 Conference Withdrawn Submission
S1 nice idea, to use a mixture of experts S2 also nice idea, to use of SVD for feature unification (Eq. 5), line 224, 225 S3 experiments show improvement on baselines S4 nice ablation study
W1. too few datasets - hard to believe that the method may work on, say, patient-doctor-diagnoses graphs, that it has never seen before. W2. Eq. 6, line243: adding the embeddings may lead to failures if the graph exhibits heterophily. Concatenation would be better - see, eg., Table 1 of the SlimG paper [Yoo et al, KDD 2023] or https://arxiv.org/pdf/2210.04081 Minor, presentation suggestions: M1. it would be nice to give the problem definition: given <??> find <??> to optimiz
* The paper highlights the key challenges of building a graph foundation model: 1/ structure heterogeneity of different graphs, 2/ feature heterogeneity of different graphs, 3/ foundation model adaptation to new domains. It is a good summary of existing challenges. * AnyGraph employs a MoE architecture to tackle the cross-domain graph heterogeneity. It proposes a AnyGraph specific method to route the experts. Specifically, the expert indicator score is computed with dot-product-based relatednes
* The proposed graph foundation model AnyGraph cannot work with heterogeneous graphs (graphs with multiple node types and edge types). * The paper is very hard to read. * Figures do not have clear captions. Here are some examples: 1. Figure 2 is not understandable. 2. Where is Figure 4(c)? What does -Feat mean in Figure 4? * Notations are not well presented and explained. Here are some examples: 1. In Section 3.1, how to get the entity representation like v_{c1}
S1. This work tackles an important problem for graph learning methods, which is to equip graph learning models with the capability to be able to efficiently adapt to a wide range of graph domains and tasks. Due to the diverse and heterogeneous nature of graph datasets (e.g., graph-structural heterogeneity and feature heterogeneity), existing approaches yet struggle to satisfy several desiderata for dealing with such real-world data, such as fast adaptation to new datasets, broad applicability, a
W1. While the most important capability of AnyGraph would be the ability to effectively generalize to new domains, it is not clear how the design of AnyGraph, in principle, can achieve strong generalization capability when given graphs from new domains. Despite strong experimental results involving cross-domain groups, it is unclear how AnyGraph can generalize to new domains that are significantly different from the domains observed during training. Given a graph from a new domain, AnyGraph will
The paper proposes a complex framework, that seems to work, but details are not enough to evaluate properly (even if I consider the appendix). The experiment section has several analyses making the paper stronger. However, important details are omitted.
Please, rewrite the abstract. According to the current abstract, the main contribution is a unified graph model, but the problem is not described. Is this unified model to generate a graph, to learn the distribution of the graph, or any other problem? Something similar happens with the introduction, where the characteristics of the model are mentioned, but not the problem. Based on page 3 (line 111), it can be deduced that the model is for prediction. Finally, in section 2, preliminaries, the pr
S1: Nice presentation. S2: their target problem is very important. S3: The ambition of exploring a graph foundation model, especially getting rid of LLMs, should be encouraged, which is rare and commendable.
W1: My 1st concern is about the semantic unifying. Different graphs usually have totally different features w.r.t semantics and dimensionality. The solution of this paper is to use SVD for dimension alignment and link prediction as a training target. However, it should be noted that even if you use SVD to achieve the same dimensionality, they are still located in different latent semantic spaces. The natural gap in the semantic space will not be narrowed with SVD tricks, and the link prediction
Code & Models
Videos
No videos yet. Explain this paper in a talk, walkthrough, or lecture? Add one.
Taxonomy
TopicsGraph Theory and Algorithms · Data Management and Algorithms