Weak convergence of the regularization path in penalized M-estimation

TL;DR

This paper establishes the weak convergence of the regularization path in penalized M-estimation, extending classical results like the CLT for the lasso to a functional setting, with broad applications.

Contribution

It provides general conditions for the weak convergence of the entire regularization path in penalized M-estimation, extending existing CLT results to a functional framework.

Findings

Weak convergence results for the regularization path.

Extension of CLT for lasso to a functional setting.

Applicability to various contrast processes.

Abstract

We consider an estimator $\hbbeta_n(t)$ defined as the element $\bphi\in\bPhi$ minimizing a contrast process $\pencontrast(\bphi, t)$ for each t. We give some general results for deriving the weak convergence of $\sqrt{n}(\hbbeta_n-\bbeta)$ in the space of bounded functions, where, for each t, $\bbeta(t)$ is the $\bphi\in\bPhi$ minimizing the limit of $\pencontrast(\bphi, t)$ as $n\to\infty$. These results are applied in the context of penalized M-estimation, that is, when $\pencontrast(\bphi, t)=M_n(\bphi)+ t J_n(\bphi)$, where $M_n$ is a usual contrast process and $J_n$ a penalty such as the $\ell^1$ norm or the squared $\ell^2$ norm. The function $\hbbeta_n$ is then called a \emph{regularization path}. For instance we show that the central limit theorem established for the lasso estimator in Knight and Fu (2000) continues to hold in a functional sense for the regularization path. Other examples include various possible contrast processes for $M_n$ such as those considered in Pollard (1985).