Neural lasso: a unifying approach of lasso and neural networks

TL;DR

This paper introduces a neural network approach to the lasso technique, aligning neural network training with statistical methods to improve variable selection, especially with small datasets, through a new optimization algorithm.

Contribution

It proposes a modified neural network training method that mimics the statistical lasso framework, introducing a new optimization algorithm for better variable selection.

Findings

The new optimization algorithm outperforms previous methods.

The neural approach achieves comparable or better accuracy.

Improved variable selection in small datasets.

Abstract

In recent years, there is a growing interest in combining techniques attributed to the areas of Statistics and Machine Learning in order to obtain the benefits of both approaches. In this article, the statistical technique lasso for variable selection is represented through a neural network. It is observed that, although both the statistical approach and its neural version have the same objective function, they differ due to their optimization. In particular, the neural version is usually optimized in one-step using a single validation set, while the statistical counterpart uses a two-step optimization based on cross-validation. The more elaborated optimization of the statistical method results in more accurate parameter estimation, especially when the training set is small. For this reason, a modification of the standard approach for training neural networks, that mimics the statistical framework, is proposed. During the development of the above modification, a new optimization algorithm for identifying the significant variables emerged. Experimental results, using synthetic and real data sets, show that this new optimization algorithm achieves better performance than any of the three previous optimization approaches.