Some sharp performance bounds for least squares regression with $L_1$ regularization

TL;DR

This paper establishes sharp bounds for L1-regularized least squares regression, extending previous results, and introduces a two-stage procedure that improves feature selection and estimation accuracy under certain conditions.

Contribution

It provides new sharp performance bounds for L1 regularization, extending prior work, and proposes a novel two-stage method with selective penalization for better results.

Findings

Sharp bounds for L1 regularization performance.

Extension of results to the Dantzig selector.

Two-stage procedure improves estimation when parameters decompose into sparse and less sparse parts.

Abstract

We derive sharp performance bounds for least squares regression with $L_1$ regularization from parameter estimation accuracy and feature selection quality perspectives. The main result proved for $L_1$ regularization extends a similar result in [Ann. Statist. 35 (2007) 2313--2351] for the Dantzig selector. It gives an affirmative answer to an open question in [Ann. Statist. 35 (2007) 2358--2364]. Moreover, the result leads to an extended view of feature selection that allows less restrictive conditions than some recent work. Based on the theoretical insights, a novel two-stage $L_1$-regularization procedure with selective penalization is analyzed. It is shown that if the target parameter vector can be decomposed as the sum of a sparse parameter vector with large coefficients and another less sparse vector with relatively small coefficients, then the two-stage procedure can lead to improved performance.