Nonuniform random feature models using derivative information

TL;DR

This paper introduces nonuniform, data-driven parameter distributions for neural network initialization that leverage derivative information, improving upon traditional uniform random feature models in regression tasks.

Contribution

It develops novel nonuniform parameter distributions based on derivative data, enhancing neural network initialization for better approximation of local derivatives.

Findings

Distributions concentrate in regions suited for local derivatives.

Sampling efficiency is improved with approximate derivative data.

Performance approaches that of optimal trained networks.

Abstract

We propose nonuniform data-driven parameter distributions for neural network initialization based on derivative data of the function to be approximated. These parameter distributions are developed in the context of non-parametric regression models based on shallow neural networks, and compare favorably to well-established uniform random feature models based on conventional weight initialization. We address the cases of Heaviside and ReLU activation functions, and their smooth approximations (sigmoid and softplus), and use recent results on the harmonic analysis and sparse representation of neural networks resulting from fully trained optimal networks. Extending analytic results that give exact representation, we obtain densities that concentrate in regions of the parameter space corresponding to neurons that are well suited to model the local derivatives of the unknown function. Based on these results, we suggest simplifications of these exact densities based on approximate derivative data in the input points that allow for very efficient sampling and lead to performance of random feature models close to optimal networks in several scenarios.