Issues with Neural Tangent Kernel Approach to Neural Networks

TL;DR

This paper critically examines the practical validity of the neural tangent kernel (NTK) equivalence theorem, revealing discrepancies between theoretical predictions and empirical results, and questioning NTK's effectiveness in modeling neural network training.

Contribution

The paper provides a rigorous re-derivation of NTK theory and empirical evidence showing the theorem's limited applicability in real neural network training scenarios.

Findings

Adding layers does not match predictor error changes predicted by NTK.

Kernel regression with Gaussian process kernels performs similarly to NTK-based regression.

The NTK equivalence theorem does not hold well in practical neural network training.

Abstract

Neural tangent kernels (NTKs) have been proposed to study the behavior of trained neural networks from the perspective of Gaussian processes. An important result in this body of work is the theorem of equivalence between a trained neural network and kernel regression with the corresponding NTK. This theorem allows for an interpretation of neural networks as special cases of kernel regression. However, does this theorem of equivalence hold in practice? In this paper, we revisit the derivation of the NTK rigorously and conduct numerical experiments to evaluate this equivalence theorem. We observe that adding a layer to a neural network and the corresponding updated NTK do not yield matching changes in the predictor error. Furthermore, we observe that kernel regression with a Gaussian process kernel in the literature that does not account for neural network training produces prediction errors very close to that of kernel regression with NTKs. These observations suggest the equivalence theorem does not hold well in practice and puts into question whether neural tangent kernels adequately address the training process of neural networks.