Sparse-Input Neural Network using Group Concave Regularization

TL;DR

This paper introduces a novel sparse-input neural network framework using group concave regularization, improving feature selection and prediction accuracy in high-dimensional data settings.

Contribution

It proposes a new regularization approach with theoretical guarantees, addressing limitations of previous methods like group LASSO in neural networks.

Findings

Effective feature selection in high-dimensional neural networks

Finite-sample guarantees for variable selection and prediction

Validated through simulations and real data applications

Abstract

Simultaneous feature selection and non-linear function estimation is challenging in modeling, especially in high-dimensional settings where the number of variables exceeds the available sample size. In this article, we investigate the problem of feature selection in neural networks. Although the group least absolute shrinkage and selection operator (LASSO) has been utilized to select variables for learning with neural networks, it tends to select unimportant variables into the model to compensate for its over-shrinkage. To overcome this limitation, we propose a framework of sparse-input neural networks using group concave regularization for feature selection in both low-dimensional and high-dimensional settings. The main idea is to apply a proper concave penalty to the $l_2$ norm of weights from all outgoing connections of each input node, and thus obtain a neural net that only uses a small subset of the original variables. In addition, we develop an effective algorithm based on backward path-wise optimization to yield stable solution paths, in order to tackle the challenge of complex optimization landscapes. We provide a rigorous theoretical analysis of the proposed framework, establishing finite-sample guarantees for both variable selection consistency and prediction accuracy. These results are supported by extensive simulation studies and real data applications, which demonstrate the finite-sample performance of the estimator in feature selection and prediction across continuous, binary, and time-to-event outcomes.