Multi-layer random features and the approximation power of neural networks

TL;DR

This paper investigates the approximation capabilities of neural networks with random weights, demonstrating that multi-layer random features can efficiently approximate functions within the RKHS defined by the NNGP, surpassing some classical bounds.

Contribution

The authors prove that neural networks with random weights can approximate functions in the NNGP RKHS and compare the efficiency of multi-layer features to Barron's theorem, providing new theoretical insights.

Findings

Multi-layer random features can approximate functions in the NNGP RKHS.

The approximation efficiency depends on the eigenvalue decay rate of the NNGP kernel.

Experiments confirm that neural networks learn target functions beyond theoretical guarantees.

Abstract

A neural architecture with randomly initialized weights, in the infinite width limit, is equivalent to a Gaussian Random Field whose covariance function is the so-called Neural Network Gaussian Process kernel (NNGP). We prove that a reproducing kernel Hilbert space (RKHS) defined by the NNGP contains only functions that can be approximated by the architecture. To achieve a certain approximation error the required number of neurons in each layer is defined by the RKHS norm of the target function. Moreover, the approximation can be constructed from a supervised dataset by a random multi-layer representation of an input vector, together with training of the last layer's weights. For a 2-layer NN and a domain equal to an $n-1$-dimensional sphere in ${\mathbb R}^n$, we compare the number of neurons required by Barron's theorem and by the multi-layer features construction. We show that if eigenvalues of the integral operator of the NNGP decay slower than $k^{-n-\frac{2}{3}}$ where $k$ is an order of an eigenvalue, then our theorem guarantees a more succinct neural network approximation than Barron's theorem. We also make some computational experiments to verify our theoretical findings. Our experiments show that realistic neural networks easily learn target functions even when both theorems do not give any guarantees.