Accurate and Scalable Graph Neural Networks via Message Invariance
Zhihao Shi, Jie Wang, Zhiwei Zhuang, Xize Liang, Bin Li, Feng Wu
TL;DR
This paper introduces TOP, a novel method for large-scale graph neural networks that achieves accurate results by transforming message passing to avoid costly neighbor computations, significantly improving speed.
Contribution
The paper proposes message invariance and topological compensation (TOP), enabling scalable GNNs by replacing expensive message passing with an invariant transformation.
Findings
TOP is significantly faster than existing methods on large graphs.
The approach maintains comparable accuracy with reduced computational cost.
Experiments on graphs with millions of nodes demonstrate scalability.
Abstract
Message passing-based graph neural networks (GNNs) have achieved great success in many real-world applications. For a sampled mini-batch of target nodes, the message passing process is divided into two parts: message passing between nodes within the batch (MP-IB) and message passing from nodes outside the batch to those within it (MP-OB). However, MP-OB recursively relies on higher-order out-of-batch neighbors, leading to an exponentially growing computational cost with respect to the number of layers. Due to the neighbor explosion, the whole message passing stores most nodes and edges on the GPU such that many GNNs are infeasible to large-scale graphs. To address this challenge, we propose an accurate and fast mini-batch approach for large graph transductive learning, namely topological compensation (TOP), which obtains the outputs of the whole message passing solely through MP-IB,…
Peer Reviews
Decision·ICLR 2025 Poster
The concept of message invariance is innovative and addresses a critical bottleneck in GNN mini-batch processing. Converting MP-OB to MP-IB is well-motivated and theoretically sound.
For large datasets, transformers based architechtures are very used for these datasets. Can your theoritical analysis be extended to GNN which doesnt follow the message passing scheme, for example GraphGPS?
- Message invariance seems like an interesting new idea. - The authors give a couple of theoretical examples for simple problems (using linear GNNs) in which message invariance holds. I understand that generalizing this to include non-linearities can be very difficult. - Their method is more computationally efficient than other baselines.
- I am unsure whether message-invariance exists. The method proposed by the authors, see for example equation 6, could also be interpreted as a graph rewiring of the subsampled nodes of the induced subgraph. Good performance could come from rewiring and alleviation of other known issues in GNNs such as over-squashing for instance, rather than due to learning message-invariance. - The large graph datasets that the authors use tend to be quite local. Rather than message-invariance, it could just b
1. The idea of using an invariant structure to approximate messages from out-of-batch nodes to in-batch nodes is unique and interesting. 2. It boasts an $\mathcal{O}(N^{-4})$ convergence rate in the number of iterations $N$, equal to node sampling methods (Section 5.3). 3. The message invariance structure is demonstrated in some dummy cases (Section 4.2), and then validated via experiments (Section 5.2). As simple as a linear approximation is competitive on several large datasets, outperformin
My rating for the work is low right now primarily because the presentation can be significantly improved. I also includes suggestions here, since I think some of them are weaknesses in the manuscript. 1. The citation (Ma & Tang, 2021), near line 46, is a book on graph deep learning, which is too general. I assume you wish to direct the reader to chapter 7 in it. It would be helpful to specify that along with the citation. 2. There seems to be some inconsistency in the definition of the edge se
Code & Models
Videos
No videos yet. Explain this paper in a talk, walkthrough, or lecture? Add one.
Taxonomy
TopicsAdvanced Graph Neural Networks · Graph Theory and Algorithms · Recommender Systems and Techniques