Selection and Estimation Optimality in High Dimensions with the TWIN Penalty

TL;DR

The paper introduces the TWIN penalty class for variable selection in high-dimensional linear models, achieving high accuracy, minimax optimality, and practical efficiency with improved performance over existing methods.

Contribution

It proposes a new TWIN penalty class with data-adaptive properties, providing theoretical guarantees and efficient algorithms for large-scale variable selection.

Findings

TWIN penalties achieve high probability of correct model selection.

TWIN penalties result in minimax optimal estimators.

TWIN outperforms standard penalties in high correlation scenarios.

Abstract

We introduce a novel class of variable selection penalties called TWIN, which provides sensible data-adaptive penalization. Under a linear sparsity regime and random Gaussian designs we show that penalties in the TWIN class have a high probability of selecting the correct model and furthermore result in minimax optimal estimators. The general shape of penalty functions in the TWIN class is the key ingredient to its desirable properties and results in improved theoretical and empirical performance over existing penalties. In this work we introduce two examples of TWIN penalties that admit simple and efficient coordinate descent algorithms, making TWIN practical in large data settings. We demonstrate in challenging and realistic simulation settings with high correlations between active and inactive variables that TWIN has high power in variable selection while controlling the number of false discoveries, outperforming standard penalties.