Asymptotic behavior of unregularized and ridge-regularized high-dimensional robust regression estimators : rigorous results

TL;DR

This paper rigorously analyzes the asymptotic behavior of high-dimensional robust regression estimators with ridge regularization, extending previous heuristic results and handling non-Gaussian design matrices.

Contribution

Provides a rigorous proof of the asymptotic behavior of ridge-regularized robust regression estimators in high dimensions, including non-Gaussian designs, building on earlier heuristic analyses.

Findings

Established asymptotic distribution of estimators

Extended results to non-Gaussian design matrices

Connected regularized and unregularized cases through limits

Abstract

We study the behavior of high-dimensional robust regression estimators in the asymptotic regime where $p/n$ tends to a finite non-zero limit. More specifically, we study ridge-regularized estimators, i.e $\widehat{\beta}=\text{argmin}_{\beta \in \mathbb{R}^p} \frac{1}{n}\sum_{i=1}^n \rho(\varepsilon_i-X_i' \beta)+\frac{\tau}{2}\lVert\beta\rVert^2$. In a recently published paper, we had developed with collaborators probabilistic heuristics to understand the asymptotic behavior of $\widehat{\beta}$. We give here a rigorous proof, properly justifying all the arguments we had given in that paper. Our proof is based on the probabilistic heuristics we had developed, and hence ideas from random matrix theory, measure concentration and convex analysis. While most the work is done for $\tau>0$, we show that under some extra assumptions on $\rho$, it is possible to recover the case $\tau=0$ as a limiting case. We require that the $X_i$'s be i.i.d with independent entries, but our proof handles the case where these entries are not Gaussian. A 2-week old paper of Donoho and Montanari [arXiv:1310.7320] studied a similar problem by a different method and with a different point of view. At this point, their interesting approach requires Gaussianity of the design matrix.