Finite Versus Infinite Neural Networks: an Empirical Study

TL;DR

This study empirically compares finite neural networks with their infinite-width kernel counterparts, revealing key differences, effects of various regularizations, and proposing best practices for kernel-based predictions, achieving state-of-the-art results.

Contribution

It provides a comprehensive empirical analysis of the correspondence between finite neural networks and infinite kernel methods, introducing improved practices and insights into their differences.

Findings

Kernel methods outperform finite fully-connected networks.

Convolutional networks outperform fully-connected networks.

NNGP kernels often outperform NT kernels.

Abstract

We perform a careful, thorough, and large scale empirical study of the correspondence between wide neural networks and kernel methods. By doing so, we resolve a variety of open questions related to the study of infinitely wide neural networks. Our experimental results include: kernel methods outperform fully-connected finite-width networks, but underperform convolutional finite width networks; neural network Gaussian process (NNGP) kernels frequently outperform neural tangent (NT) kernels; centered and ensembled finite networks have reduced posterior variance and behave more similarly to infinite networks; weight decay and the use of a large learning rate break the correspondence between finite and infinite networks; the NTK parameterization outperforms the standard parameterization for finite width networks; diagonal regularization of kernels acts similarly to early stopping; floating point precision limits kernel performance beyond a critical dataset size; regularized ZCA whitening improves accuracy; finite network performance depends non-monotonically on width in ways not captured by double descent phenomena; equivariance of CNNs is only beneficial for narrow networks far from the kernel regime. Our experiments additionally motivate an improved layer-wise scaling for weight decay which improves generalization in finite-width networks. Finally, we develop improved best practices for using NNGP and NT kernels for prediction, including a novel ensembling technique. Using these best practices we achieve state-of-the-art results on CIFAR-10 classification for kernels corresponding to each architecture class we consider.