An adaptive Ridge procedure for L0 regularization

TL;DR

This paper introduces an adaptive Ridge (AR) method for L0 regularization, offering a computationally feasible alternative to non-convex optimization in high-dimensional variable selection, with theoretical analysis and practical applications.

Contribution

The paper proposes the adaptive Ridge procedure as a novel approach to approximate L0 regularization, extending adaptive lasso ideas with weighted Ridge problems and analyzing its properties.

Findings

AR mimics adaptive lasso but uses Ridge problems

Theoretical properties are established for orthogonal linear regression

Extensive simulations demonstrate AR's performance in non-orthogonal cases

Abstract

Penalized selection criteria like AIC or BIC are among the most popular methods for variable selection. Their theoretical properties have been studied intensively and are well understood, but making use of them in case of high-dimensional data is difficult due to the non-convex optimization problem induced by L0 penalties. An elegant solution to this problem is provided by the multi-step adaptive lasso, where iteratively weighted lasso problems are solved, whose weights are updated in such a way that the procedure converges towards selection with L0 penalties. In this paper we introduce an adaptive ridge procedure (AR) which mimics the adaptive lasso, but is based on weighted Ridge problems. After introducing AR its theoretical properties are studied in the particular case of orthogonal linear regression. For the non-orthogonal case extensive simulations are performed to assess the performance of AR. In case of Poisson regression and logistic regression it is illustrated how the iterative procedure of AR can be combined with iterative maximization procedures. The paper ends with an efficient implementation of AR in the context of least-squares segmentation.