Efficient and Effective Implicit Dynamic Graph Neural Network

TL;DR

This paper introduces IDGNN, a novel implicit dynamic graph neural network that guarantees fixed-point solutions and employs a bilevel optimization-based training algorithm, achieving significant speed-ups over traditional methods while maintaining high performance.

Contribution

It is the first implicit neural network model designed for dynamic graphs, with a theoretically guaranteed fixed-point representation and an efficient training algorithm.

Findings

IDGNN outperforms state-of-the-art baselines in real-world tasks.

The proposed method achieves up to 1600x speed-up.

The model maintains performance comparable to iterative algorithms.

Abstract

Implicit graph neural networks have gained popularity in recent years as they capture long-range dependencies while improving predictive performance in static graphs. Despite the tussle between performance degradation due to the oversmoothing of learned embeddings and long-range dependency being more pronounced in dynamic graphs, as features are aggregated both across neighborhood and time, no prior work has proposed an implicit graph neural model in a dynamic setting. In this paper, we present Implicit Dynamic Graph Neural Network (IDGNN) a novel implicit neural network for dynamic graphs which is the first of its kind. A key characteristic of IDGNN is that it demonstrably is well-posed, i.e., it is theoretically guaranteed to have a fixed-point representation. We then demonstrate that the standard iterative algorithm often used to train implicit models is computationally expensive in our dynamic setting as it involves computing gradients, which themselves have to be estimated in an iterative manner. To overcome this, we pose an equivalent bilevel optimization problem and propose an efficient single-loop training algorithm that avoids iterative computation by maintaining moving averages of key components of the gradients. We conduct extensive experiments on real-world datasets on both classification and regression tasks to demonstrate the superiority of our approach over the state-of-the-art baselines. We also demonstrate that our bi-level optimization framework maintains the performance of the expensive iterative algorithm while obtaining up to \textbf{1600x} speed-up.