An Empirical Analysis of the Laplace and Neural Tangent Kernels

TL;DR

This paper empirically investigates the relationship between the Laplace and neural tangent kernels, demonstrating their practical equivalence through kernel matching and regression experiments.

Contribution

It provides the first detailed empirical analysis comparing the Laplace and neural tangent kernels, including kernel matching and regression performance evaluations.

Findings

The kernels can be matched exactly in certain settings.

They exhibit similar posterior distributions in Gaussian process regression.

Experimental results support their practical equivalence.

Abstract

The neural tangent kernel is a kernel function defined over the parameter distribution of an infinite width neural network. Despite the impracticality of this limit, the neural tangent kernel has allowed for a more direct study of neural networks and a gaze through the veil of their black box. More recently, it has been shown theoretically that the Laplace kernel and neural tangent kernel share the same reproducing kernel Hilbert space in the space of $\mathbb{S}^{d-1}$ alluding to their equivalence. In this work, we analyze the practical equivalence of the two kernels. We first do so by matching the kernels exactly and then by matching posteriors of a Gaussian process. Moreover, we analyze the kernels in $\mathbb{R}^d$ and experiment with them in the task of regression.