Is Solving Graph Neural Tangent Kernel Equivalent to Training Graph Neural Network?

TL;DR

This paper establishes the theoretical equivalence between training infinite-wide graph neural networks and solving their corresponding Graph Neural Tangent Kernel (GNTK) regression, extending the understanding of GNTK's capabilities.

Contribution

The paper proves the equivalence between GNTK regression and training GNNs for both graph-level and node-level tasks, and introduces the first GNTK formulation for node-level regression.

Findings

Proves GNTK and GNN training equivalence for graph-level regression.

Introduces the first GNTK formulation for node-level regression.

Establishes theoretical foundations for GNTK's use in graph learning.

Abstract

A rising trend in theoretical deep learning is to understand why deep learning works through Neural Tangent Kernel (NTK) [jgh18], a kernel method that is equivalent to using gradient descent to train a multi-layer infinitely-wide neural network. NTK is a major step forward in the theoretical deep learning because it allows researchers to use traditional mathematical tools to analyze properties of deep neural networks and to explain various neural network techniques from a theoretical view. A natural extension of NTK on graph learning is \textit{Graph Neural Tangent Kernel (GNTK)}, and researchers have already provide GNTK formulation for graph-level regression and show empirically that this kernel method can achieve similar accuracy as GNNs on various bioinformatics datasets [dhs+19]. The remaining question now is whether solving GNTK regression is equivalent to training an infinite-wide multi-layer GNN using gradient descent. In this paper, we provide three new theoretical results. First, we formally prove this equivalence for graph-level regression. Second, we present the first GNTK formulation for node-level regression. Finally, we prove the equivalence for node-level regression.