Higher Order Refinements by Bootstrap in Lasso and other Penalized Regression Methods

TL;DR

This paper investigates the theoretical properties of bootstrap methods in high-dimensional penalized regression, especially focusing on the Lasso, and demonstrates their second order correctness regardless of the estimator class.

Contribution

It classifies penalized regression methods based on oracle properties and proves bootstrap methods are second order correct for all such estimators.

Findings

Bootstrap methods are second order correct for all penalized estimators.

Lasso is variable selection consistent but does not satisfy the oracle property.

The paper provides a theoretical framework for bootstrap in high-dimensional regression.

Abstract

Selection of important covariates and to drop the unimportant ones from a high-dimensional regression model is a long standing problem and hence have received lots of attention in the last two decades. After selecting the correct model, it is also important to properly estimate the existing parameters corresponding to important covariates. In this spirit, Fan and Li (2001) proposed Oracle property as a desired feature of a variable selection method. Oracle property has two parts; one is the variable selection consistency (VSC) and the other one is the asymptotic normality. Keeping VSC fixed and making the other part stronger, Fan and Lv (2008) introduced the strong oracle property. In this paper, we consider different penalized regression techniques which are VSC and classify those based on oracle and strong oracle property. We show that both the residual and the perturbation bootstrap methods are second order correct for any penalized estimator irrespective of its class. Most interesting of all is the Lasso, introduced by Tibshirani (1996). Although Lasso is VSC, it is not asymptotically normal and hence fails to satisfy the oracle property.