On neural network kernels and the storage capacity problem

TL;DR

This paper establishes a theoretical link between the storage capacity of wide two-layer neural networks and their kernel limits, revealing that the effective order parameter is equivalent to the Neural Network Gaussian Process Kernel, thus connecting expressivity and trainability.

Contribution

It uncovers a direct equivalence between the effective order parameter in statistical mechanics and the Neural Network Gaussian Process Kernel in wide neural networks.

Findings

Effective order parameter equals the Neural Network Gaussian Process Kernel.

Connects expressivity with kernel limits in wide neural networks.

Provides a theoretical foundation linking statistical mechanics and kernel methods.

Abstract

In this short note, we reify the connection between work on the storage capacity problem in wide two-layer treelike neural networks and the rapidly-growing body of literature on kernel limits of wide neural networks. Concretely, we observe that the "effective order parameter" studied in the statistical mechanics literature is exactly equivalent to the infinite-width Neural Network Gaussian Process Kernel. This correspondence connects the expressivity and trainability of wide two-layer neural networks.