Gradient Kernel Regression

TL;DR

This paper demonstrates that kernel regression using the neural tangent kernel achieves accuracy independent of the underlying network's training quality, revealing a surprising property of the kernel's behavior.

Contribution

It introduces a novel insight that neural tangent kernel-based regression's accuracy does not depend on the underlying network's training accuracy.

Findings

Kernel regression with neural tangent kernel is accuracy-independent of network training quality.

The neural tangent kernel's properties lead to surprising robustness in regression tasks.

This work challenges assumptions about the relationship between network training and kernel regression performance.

Abstract

In this article a surprising result is demonstrated using the neural tangent kernel. This kernel is defined as the inner product of the vector of the gradient of an underlying model evaluated at training points. This kernel is used to perform kernel regression. The surprising thing is that the accuracy of that regression is independent of the accuracy of the underlying network.