On the Distribution of Penalized Maximum Likelihood Estimators: The LASSO, SCAD, and Thresholding

TL;DR

This paper analyzes the distributional properties of LASSO, SCAD, and thresholding estimators, revealing their nonnormality and convergence rates in finite and large samples, with implications for statistical inference.

Contribution

It provides the first comprehensive analysis of the asymptotic and finite-sample distributions of these penalized estimators, including convergence rates and an impossibility result.

Findings

Estimators are typically highly nonnormal regardless of tuning.

Convergence rate is slower than 1/√n when tuned for consistent model selection.

An impossibility result for estimating the distribution function of these estimators.

Abstract

We study the distributions of the LASSO, SCAD, and thresholding estimators, in finite samples and in the large-sample limit. The asymptotic distributions are derived for both the case where the estimators are tuned to perform consistent model selection and for the case where the estimators are tuned to perform conservative model selection. Our findings complement those of Knight and Fu (2000) and Fan and Li (2001). We show that the distributions are typically highly nonnormal regardless of how the estimator is tuned, and that this property persists in large samples. The uniform convergence rate of these estimators is also obtained, and is shown to be slower than 1/root(n) in case the estimator is tuned to perform consistent model selection. An impossibility result regarding estimation of the estimators' distribution function is also provided.