Asymptotics of constrained $M$-estimation under convexity

TL;DR

This paper develops an asymptotic theory for convex M-estimators, including non-differentiable cases, under convex constraints, revealing how their distributions depend on loss functions and boundary structures, with applications in robust estimation.

Contribution

It introduces a novel asymptotic framework for convex M-estimation without differentiability, extending to U-estimators and diverse applications.

Findings

Asymptotic distributions depend on loss and boundary structure.

Extended results to U-estimators based on U-statistics.

Applicable to robust location, scatter, and depth estimation.

Abstract

M-estimation, aka empirical risk minimization, is at the heart of statistics and machine learning: Classification, regression, location estimation, etc. Asymptotic theory is well understood when the loss satisfies some smoothness assumptions and its derivatives are dominated locally. However, these conditions are typically technical and can be too restrictive or heavy to check. Here, we consider the case of a convex loss function, which may not even be differentiable: We establish an asymptotic theory for M-estimation with convex loss (which needs not be differentiable) under convex constraints. We show that the asymptotic distributions of the corresponding M-estimators depend on an interplay between the loss function and the boundary structure of the set of constraints. We extend our results to U-estimators, building on the asymptotic theory of U-statistics. Applications of our work include, among other, robust location/scatter estimation, estimation of deepest points relative to depth functions such as Oja's depth, etc.