Every Model Learned by Gradient Descent Is Approximately a Kernel Machine

TL;DR

This paper demonstrates that deep neural networks trained with gradient descent are approximately equivalent to kernel machines, providing new insights into their interpretability and potential for developing improved learning algorithms.

Contribution

It reveals that standard deep networks are mathematically similar to kernel machines, linking deep learning to classical kernel methods and enhancing interpretability.

Findings

Deep networks approximate kernel machines when trained with gradient descent.

Network weights can be viewed as a superposition of training examples.

Incorporating target knowledge into the kernel improves learning.

Abstract

Deep learning's successes are often attributed to its ability to automatically discover new representations of the data, rather than relying on handcrafted features like other learning methods. We show, however, that deep networks learned by the standard gradient descent algorithm are in fact mathematically approximately equivalent to kernel machines, a learning method that simply memorizes the data and uses it directly for prediction via a similarity function (the kernel). This greatly enhances the interpretability of deep network weights, by elucidating that they are effectively a superposition of the training examples. The network architecture incorporates knowledge of the target function into the kernel. This improved understanding should lead to better learning algorithms.