IGNN-Solver: A Graph Neural Solver for Implicit Graph Neural Networks
Junchao Lin, Zenan Ling, Zhanbo Feng, Jingwen Xu, Minxuan Liao, Feng, Zhou, Tianqi Hou, Zhenyu Liao, Robert C. Qiu
TL;DR
The paper introduces IGNN-Solver, a novel method that accelerates implicit graph neural networks using Anderson Acceleration and graph-specific optimizations, enabling faster inference on large-scale graphs without losing accuracy.
Contribution
It proposes a new graph neural solver that significantly speeds up IGNNs by combining Anderson Acceleration with graph-specific sparsification and compression techniques.
Findings
Achieves 1.5x to 8x speedup in inference time.
Maintains accuracy while improving scalability.
Effective on both small and large-scale graph tasks.
Abstract
Implicit graph neural networks (IGNNs), which exhibit strong expressive power with a single layer, have recently demonstrated remarkable performance in capturing long-range dependencies (LRD) in underlying graphs while effectively mitigating the over-smoothing problem. However, IGNNs rely on computationally expensive fixed-point iterations, which lead to significant speed and scalability limitations, hindering their application to large-scale graphs. To achieve fast fixed-point solving for IGNNs, we propose a novel graph neural solver, IGNN-Solver, which leverages the generalized Anderson Acceleration method, parameterized by a tiny GNN, and learns iterative updates as a graph-dependent temporal process. To improve effectiveness on large-scale graph tasks, we further integrate sparsification and storage compression methods, specifically tailored for the IGNN-Solver, into its design.…
Peer Reviews
Decision·ICLR 2025 Conference Withdrawn Submission
1. This paper tries to answer a very important question: how to accelerate Implicit Graph Neural Networks (IGNNs), which is of interest in the community. IGNNs have some advantages over traditional GNNs, while IGNNs suffer from slow training and inference speed. And this hinders the usage of IGNNs in many applications, espcially when graphs are large. 2. The high-level idea of the proposed solver is cleary demonstrated in Figure 2, which make it easy to understand. 3. Although the novelty of t
1. More deeper analysis on why the proposed solver can be faster than others. In my view, the speedup comes from the less number of iterations required. I think that would be better if the authors can provided some theoretical analysis. If theoreical anlaysis is difficult to have, I would like to see some empirical evidences on how many iterations the proposed solver needs vs the traditional solver needs. 2. The high-level descriptions on the method (RPI-Graph) used for graph sparsification are
1. The paper demonstrates that IGNN-Solver can achieve a significant speedup in inference time compared to regular IGNNs, and the additional training cost of the IGNN-Solver is minimal.
1. Current background lacks coverage of recent advances, such as: Method [1], which models IGNN as a bilevel optimization problem, achieving significant speedups. Method [2], a scalable implicit model with higher accuracy on the ogbn-arxiv dataset. Given [2] is already cited, consider a comparison to highlight IGNN-Solver's distinct advantages in context. 2. Since efficiency is a core advantage of IGNN-Solver, it is crucial to benchmark its runtime against multiple existing methods beyond the b
IGNN-Solver introduces a low overhead in the training procedure (1% - 2% of the total training time).
1. The expreriment results in Table 1 cannot convince me. At least, the baselines for ogbn-arxiv and ogbn-products is severely sandbagged. For example, on ogb learderboard, GCNII is 72.74 for ogbn-arxiv, and GCN is 75.64 on ogbn-products. These gaps are more than 4% compared with the number reported in the paper. 2. There is no study on how accurate the alpha predicted by the tiny GNN model.
1. The paper is well-written, clear, and easy to comprehend. 2. The experimental results show the proposed neural solver can achieve significant acceleration for IGNN, yielding several-fold speedups without compromising predictive performance on several benchmark graph datasets.
1. **Lack of Theoretical Justifications:** The paper does not provide sufficient theoretical underpinnings for the proposed fixed-point neural solver. Notably, it remains unclear if the fixed-point equation is well-posed, and there are no convergence guarantees for the solver. 2. **Uncertain Effectiveness of the Learnable Initializer:** Without guarantees of convergence, it is ambiguous whether the learnable initializer can indeed reduce the number of iterations required, thus raising questions
1. Due to the implicit layer of the IGNN, formulated as a fixed-point equation, it can access infinite hops of neighbors implicitly. This enables IGNN to address the long-standing over-smoothing and long-range dependency issues that have plagued explicit GNNs, preventing them from becoming deeper and larger. Therefore, optimizing the computational burden of IGNN, which has stronger scalability, is a potentially promising topic in the journey towards larger graph models. I acknowledge the signifi
1. Given that the IGNN algorithm can implicitly capture long-range dependencies, I had hoped that the IGNN-Solver would show a greater advantage over explicit GNNs on large datasets. However, on the ogbn-arxiv and ogbn-products datasets, the IGNN-Solver did not demonstrate a significant performance improvement over explicit GNNs. I would like to see not only the comparison of inference times between the IGNN-Solver and traditional IGNN, but also a comparison of the training and inference speeds
Videos
No videos yet. Explain this paper in a talk, walkthrough, or lecture? Add one.
Taxonomy
TopicsNeural Networks and Applications
MethodsSPEED: Separable Pyramidal Pooling EncodEr-Decoder for Real-Time Monocular Depth Estimation on Low-Resource Settings