Deterministic Inequalities for Smooth M-estimators

TL;DR

This paper develops deterministic inequalities for smooth M-estimators using the Banach fixed point theorem, enabling finite sample analysis and applications in high-dimensional statistics, post-selection inference, and various classical models.

Contribution

It introduces a novel deterministic approach to analyze M-estimators, extending classical asymptotic results to finite samples and complex models.

Findings

Deterministic inequalities derived via Banach fixed point theorem

Applications to cross-validation, marginal screening, and post-selection inference

Extensions to non-smooth and constrained optimization problems

Abstract

Ever since the proof of asymptotic normality of maximum likelihood estimator by Cramer (1946), it has been understood that a basic technique of the Taylor series expansion suffices for asymptotics of $M$-estimators with smooth/differentiable loss function. Although the Taylor series expansion is a purely deterministic tool, the realization that the asymptotic normality results can also be made deterministic (and so finite sample) received far less attention. With the advent of big data and high-dimensional statistics, the need for finite sample results has increased. In this paper, we use the (well-known) Banach fixed point theorem to derive various deterministic inequalities that lead to the classical results when studied under randomness. In addition, we provide applications of these deterministic inequalities for crossvalidation/subsampling, marginal screening and uniform-in-submodel results that are very useful for post-selection inference and in the study of post-regularization estimators. Our results apply to many classical estimators, in particular, generalized linear models, non-linear regression and cox proportional hazards model. Extensions to non-smooth and constrained problems are also discussed.