An Exact Kernel Equivalence for Finite Classification Models

TL;DR

This paper derives an exact kernel representation for finite neural network classifiers trained with gradient descent, enabling precise analysis of their behavior and generalization, surpassing previous approximate kernel methods.

Contribution

It presents the first exact kernel equivalence for finite neural networks, allowing precise theoretical and practical analysis of neural network predictions.

Findings

Kernel can be computed for realistic networks up to machine precision

Exact kernel provides insights into neural network generalization

Comparison shows approximation errors relative to NTK

Abstract

We explore the equivalence between neural networks and kernel methods by deriving the first exact representation of any finite-size parametric classification model trained with gradient descent as a kernel machine. We compare our exact representation to the well-known Neural Tangent Kernel (NTK) and discuss approximation error relative to the NTK and other non-exact path kernel formulations. We experimentally demonstrate that the kernel can be computed for realistic networks up to machine precision. We use this exact kernel to show that our theoretical contribution can provide useful insights into the predictions made by neural networks, particularly the way in which they generalize.