On model selection consistency of regularized M-estimators

TL;DR

This paper develops a general framework to analyze the conditions under which regularized M-estimators reliably identify the correct low-dimensional model structure in high-dimensional settings.

Contribution

It introduces the concepts of geometric decomposability and irrepresentability, providing a unified approach to establish consistency and model selection consistency.

Findings

Identifies key properties ensuring estimator consistency.

Provides conditions for model selection consistency.

Applies framework to specific statistical learning cases.

Abstract

Regularized M-estimators are used in diverse areas of science and engineering to fit high-dimensional models with some low-dimensional structure. Usually the low-dimensional structure is encoded by the presence of the (unknown) parameters in some low-dimensional model subspace. In such settings, it is desirable for estimates of the model parameters to be \emph{model selection consistent}: the estimates also fall in the model subspace. We develop a general framework for establishing consistency and model selection consistency of regularized M-estimators and show how it applies to some special cases of interest in statistical learning. Our analysis identifies two key properties of regularized M-estimators, referred to as geometric decomposability and irrepresentability, that ensure the estimators are consistent and model selection consistent.