Asymptotic linear expansion of regularized M-estimators

TL;DR

This paper investigates the conditions under which regularized M-estimators in high-dimensional regression are asymptotically linear, enabling simpler analysis and application in machine learning algorithms.

Contribution

It establishes criteria for the compact differentiability of M-functionals, facilitating asymptotic linear expansion of regularized estimators in high-dimensional settings.

Findings

Conditions for compact differentiability of M-functionals

Asymptotic linear expansion applies to certain machine learning algorithms

Simplifies the analysis of regularized estimators in high-dimensional regression

Abstract

Parametric high-dimensional regression analysis requires the usage of regularization terms to get interpretable models. The respective estimators can be regarded as regularized M-functionals which are naturally highly nonlinear. We study under which conditions these M-functionals are compactly differentiable, so that the corresponding estimators admit an asymptotically linear expansion. In a one-step construction, for a suitably consistent starting estimator, this linearization replaces solving optimization problems by evaluating the corresponding influence curves at the given data points. We show under which conditions the asymptotic linear expansion is valid and provide concrete examples of machine learning algorithms that fit into this framework.