New Insights into Graph Convolutional Networks using Neural Tangent Kernels

TL;DR

This paper uses Neural Tangent Kernels to analyze and improve the understanding of Graph Convolutional Networks, revealing how normalization affects depth performance and proposing NTK as a hyper-parameter free surrogate model.

Contribution

It derives NTKs for wide GCNs, explains depth performance issues, and introduces NTK as an efficient surrogate for GCNs that is insensitive to hyper-parameter tuning.

Findings

Normalized NTKs mitigate depth-related performance degradation.

NTKs accurately predict GCN behavior across different architectures.

Surrogate NTKs reduce the need for hyper-parameter tuning.

Abstract

Graph Convolutional Networks (GCNs) have emerged as powerful tools for learning on network structured data. Although empirically successful, GCNs exhibit certain behaviour that has no rigorous explanation -- for instance, the performance of GCNs significantly degrades with increasing network depth, whereas it improves marginally with depth using skip connections. This paper focuses on semi-supervised learning on graphs, and explains the above observations through the lens of Neural Tangent Kernels (NTKs). We derive NTKs corresponding to infinitely wide GCNs (with and without skip connections). Subsequently, we use the derived NTKs to identify that, with suitable normalisation, network depth does not always drastically reduce the performance of GCNs -- a fact that we also validate through extensive simulation. Furthermore, we propose NTK as an efficient `surrogate model' for GCNs that does not suffer from performance fluctuations due to hyper-parameter tuning since it is a hyper-parameter free deterministic kernel. The efficacy of this idea is demonstrated through a comparison of different skip connections for GCNs using the surrogate NTKs.