Selection by Partitioning the Solution Paths

TL;DR

This paper proposes a novel feature selection method that leverages entire solution paths of penalized likelihood models, improving accuracy and applicability to various penalties, with strong theoretical backing and empirical validation.

Contribution

It introduces a new approach that uses the full solution path for feature selection, applicable to convex penalties, with weaker theoretical conditions than existing methods.

Findings

Improved feature selection accuracy over single-parameter methods.

Applicable to ridge and other convex penalties.

Validated through simulations and real data.

Abstract

The performance of penalized likelihood approaches depends profoundly on the selection of the tuning parameter; however, there is no commonly agreed-upon criterion for choosing the tuning parameter. Moreover, penalized likelihood estimation based on a single value of the tuning parameter suffers from several drawbacks. This article introduces a novel approach for feature selection based on the entire solution paths rather than the choice of a single tuning parameter, which significantly improves the accuracy of the selection. Moreover, the approach allows for feature selection using ridge or other strictly convex penalties. The key idea is to classify variables as relevant or irrelevant at each tuning parameter and then to select all of the variables which have been classified as relevant at least once. We establish the theoretical properties of the method, which requires significantly weaker conditions than existing methods in the literature. We also illustrate the advantages of the proposed approach with simulation studies and a data example.