Bayes-optimal Learning of Deep Random Networks of Extensive-width

TL;DR

This paper derives a closed-form expression for the Bayes-optimal test error in learning deep, wide neural networks with Gaussian weights, comparing it with ridge, kernel, and neural network methods in high-dimensional regimes.

Contribution

It provides the first analytical characterization of the Bayes-optimal error for deep extensive-width networks and compares it with classical regression and kernel methods.

Findings

Bayes-optimal test error derived for deep wide networks.

Ridge and kernel regression achieve Bayes-optimal performance.

Neural networks outperform classical methods when samples grow faster than input dimension.

Abstract

We consider the problem of learning a target function corresponding to a deep, extensive-width, non-linear neural network with random Gaussian weights. We consider the asymptotic limit where the number of samples, the input dimension and the network width are proportionally large. We propose a closed-form expression for the Bayes-optimal test error, for regression and classification tasks. We further compute closed-form expressions for the test errors of ridge regression, kernel and random features regression. We find, in particular, that optimally regularized ridge regression, as well as kernel regression, achieve Bayes-optimal performances, while the logistic loss yields a near-optimal test error for classification. We further show numerically that when the number of samples grows faster than the dimension, ridge and kernel methods become suboptimal, while neural networks achieve test error close to zero from quadratically many samples.