LASSO, Iterative Feature Selection and the Correlation Selector: Oracle Inequalities and Numerical Performances

TL;DR

This paper introduces a unified family of algorithms for regression that includes well-known methods like LASSO and Dantzig selector, providing theoretical guarantees and comparing their numerical performance.

Contribution

It unifies existing regression algorithms into a single family and introduces the Correlation Selector, with proven oracle inequalities and performance comparisons.

Findings

Oracle inequalities established for IFS, LASSO, and Correlation Selector.

Numerical performance comparison on a toy example.

Correlation Selector performs competitively with existing methods.

Abstract

We propose a general family of algorithms for regression estimation with quadratic loss. Our algorithms are able to select relevant functions into a large dictionary. We prove that a lot of algorithms that have already been studied for this task (LASSO and Group LASSO, Dantzig selector, Iterative Feature Selection, among others) belong to our family, and exhibit another particular member of this family that we call Correlation Selector in this paper. Using general properties of our family of algorithm we prove oracle inequalities for IFS, for the LASSO and for the Correlation Selector, and compare numerical performances of these estimators on a toy example.