A phase transition for finding needles in nonlinear haystacks with LASSO artificial neural networks

TL;DR

This paper introduces a novel neural network approach with a hyperparameter-driven phase transition phenomenon for identifying important features in nonlinear data, inspired by LASSO's success in linear settings.

Contribution

It develops a new ANN learner that exhibits a phase transition for feature recovery, along with a generalized threshold for parameter selection and a warm-start algorithm for high-dimensional optimization.

Findings

The ANN learner shows a phase transition in needle recovery probability.

The generalized universal threshold improves hyperparameter selection.

Monte Carlo simulations confirm the approach's effectiveness.

Abstract

To fit sparse linear associations, a LASSO sparsity inducing penalty with a single hyperparameter provably allows to recover the important features (needles) with high probability in certain regimes even if the sample size is smaller than the dimension of the input vector (haystack). More recently learners known as artificial neural networks (ANN) have shown great successes in many machine learning tasks, in particular fitting nonlinear associations. Small learning rate, stochastic gradient descent algorithm and large training set help to cope with the explosion in the number of parameters present in deep neural networks. Yet few ANN learners have been developed and studied to find needles in nonlinear haystacks. Driven by a single hyperparameter, our ANN learner, like for sparse linear associations, exhibits a phase transition in the probability of retrieving the needles, which we do not observe with other ANN learners. To select our penalty parameter, we generalize the universal threshold of Donoho and Johnstone (1994) which is a better rule than the conservative (too many false detections) and expensive cross-validation. In the spirit of simulated annealing, we propose a warm-start sparsity inducing algorithm to solve the high-dimensional, non-convex and non-differentiable optimization problem. We perform precise Monte Carlo simulations to show the effectiveness of our approach.