What needles do sparse neural networks find in nonlinear haystacks

TL;DR

This paper develops a theoretical method for selecting sparsity-inducing penalties in neural networks without cross-validation, based on bounding the gradient norm, and demonstrates its effectiveness through simulations.

Contribution

It generalizes the universal threshold concept to nonlinear neural networks, enabling penalty selection without data splitting.

Findings

The proposed method effectively selects penalties in simulations.

It extends the universal threshold to nonlinear neural network training.

Numerical results confirm the approach's practicality and accuracy.

Abstract

Using a sparsity inducing penalty in artificial neural networks (ANNs) avoids over-fitting, especially in situations where noise is high and the training set is small in comparison to the number of features. For linear models, such an approach provably also recovers the important features with high probability in regimes for a well-chosen penalty parameter. The typical way of setting the penalty parameter is by splitting the data set and performing the cross-validation, which is (1) computationally expensive and (2) not desirable when the data set is already small to be further split (for example, whole-genome sequence data). In this study, we establish the theoretical foundation to select the penalty parameter without cross-validation based on bounding with a high probability the infinite norm of the gradient of the loss function at zero under the zero-feature assumption. Our approach is a generalization of the universal threshold of Donoho and Johnstone (1994) to nonlinear ANN learning. We perform a set of comprehensive Monte Carlo simulations on a simple model, and the numerical results show the effectiveness of the proposed approach.