Smaller generalization error derived for a deep residual neural network compared to shallow networks

TL;DR

This paper derives a smaller generalization error bound for deep residual neural networks using optimal random Fourier feature distributions, outperforming shallow networks and leading to a new training method with promising experimental results.

Contribution

It introduces an optimal frequency distribution for deep residual networks' random Fourier features, reducing generalization error compared to shallow networks and informing a novel training algorithm.

Findings

Smaller generalization error bound for deep residual networks.

Optimal frequency distribution derived for random Fourier features.

New training method demonstrated promising performance.

Abstract

Estimates of the generalization error are proved for a residual neural network with $L$ random Fourier features layers $\bar z_{\ell+1}=\bar z_\ell + \mathrm{Re}\sum_{k=1}^K\bar b_{\ell k}e^{\mathrm{i}\omega_{\ell k}\bar z_\ell}+ \mathrm{Re}\sum_{k=1}^K\bar c_{\ell k}e^{\mathrm{i}\omega'_{\ell k}\cdot x}$. An optimal distribution for the frequencies $(\omega_{\ell k},\omega'_{\ell k})$ of the random Fourier features $e^{\mathrm{i}\omega_{\ell k}\bar z_\ell}$ and $e^{\mathrm{i}\omega'_{\ell k}\cdot x}$ is derived. This derivation is based on the corresponding generalization error for the approximation of the function values $f(x)$. The generalization error turns out to be smaller than the estimate ${\|\hat f\|^2_{L^1(\mathbb{R}^d)}}/{(KL)}$ of the generalization error for random Fourier features with one hidden layer and the same total number of nodes $KL$, in the case the $L^\infty$-norm of $f$ is much less than the $L^1$-norm of its Fourier transform $\hat f$. This understanding of an optimal distribution for random features is used to construct a new training method for a deep residual network. Promising performance of the proposed new algorithm is demonstrated in computational experiments.