Asymptotics For High Dimensional Regression M-Estimates: Fixed Design Results

TL;DR

This paper derives the asymptotic distribution of high-dimensional regression M-estimates with fixed design matrices, demonstrating normality under certain conditions as the number of covariates grows proportionally with the sample size.

Contribution

It establishes coordinate-wise asymptotic normality for regression M-estimates in high dimensions with fixed design matrices, using novel proof techniques and broad regularity conditions.

Findings

Asymptotic normality holds for a broad class of design matrices.

Counterexample shows assumptions are necessary.

Numerical experiments support theoretical results.

Abstract

We investigate the asymptotic distributions of coordinates of regression M-estimates in the moderate $p/n$ regime, where the number of covariates $p$ grows proportionally with the sample size $n$. Under appropriate regularity conditions, we establish the coordinate-wise asymptotic normality of regression M-estimates assuming a fixed-design matrix. Our proof is based on the second-order Poincar\'{e} inequality (Chatterjee, 2009) and leave-one-out analysis (El Karoui et al., 2011). Some relevant examples are indicated to show that our regularity conditions are satisfied by a broad class of design matrices. We also show a counterexample, namely the ANOVA-type design, to emphasize that the technical assumptions are not just artifacts of the proof. Finally, the numerical experiments confirm and complement our theoretical results.