Generic chaining and the l1-penalty

TL;DR

This paper uses Talagrand's generic chaining to determine the tuning parameter for l1-penalized M-estimation in complex, high-dimensional, nonlinear models, aligning it with standard Lasso practices.

Contribution

It applies the generic chaining technique to high-dimensional nonlinear models, deriving a tuning parameter choice similar to that in linear models.

Findings

Derives a tuning parameter $oxed{ extstyle o ext{const} imes rac{ ext{sqrt}( ext{log} p)}{ ext{sqrt} n}}$ for complex models.

Shows the generic chaining approach extends beyond linear models to nonlinear, high-dimensional settings.

Provides theoretical justification for the standard $oxed{ extstyle o ext{sqrt}( ext{log} p / n)}$ penalty choice.

Abstract

We address the choice of the tuning parameter $\lambda$ in $\ell_1$-penalized M-estimation. Our main concern is models which are highly nonlinear, such as the Gaussian mixture model. The number of parameters $p$ is moreover large, possibly larger than the number of observations $n$. The generic chaining technique of Talagrand[2005] is tailored for this problem. It leads to the choice $\lambda \asymp \sqrt {\log p / n}$, as in the standard Lasso procedure (which concerns the linear model and least squares loss).