Retire: Robust Expectile Regression in High Dimensions

TL;DR

This paper introduces a robust expectile regression method for high-dimensional data that reduces bias and has strong theoretical properties, outperforming existing methods in simulations.

Contribution

The paper proposes a novel penalized robust expectile regression method with iteratively reweighted $ ext{l}_1$-penalization, providing theoretical guarantees and efficient algorithms for high-dimensional settings.

Findings

The method achieves oracle convergence rates after a logarithmic number of iterations.

It is computationally efficient using a semismooth Newton coordinate descent algorithm.

Numerical studies show it outperforms non-robust and quantile regression methods.

Abstract

High-dimensional data can often display heterogeneity due to heteroscedastic variance or inhomogeneous covariate effects. Penalized quantile and expectile regression methods offer useful tools to detect heteroscedasticity in high-dimensional data. The former is computationally challenging due to the non-smooth nature of the check loss, and the latter is sensitive to heavy-tailed error distributions. In this paper, we propose and study (penalized) robust expectile regression (retire), with a focus on iteratively reweighted $\ell_1$-penalization which reduces the estimation bias from $\ell_1$-penalization and leads to oracle properties. Theoretically, we establish the statistical properties of the retire estimator under two regimes: (i) low-dimensional regime in which $d \ll n$; (ii) high-dimensional regime in which $s\ll n\ll d$ with $s$ denoting the number of significant predictors. In the high-dimensional setting, we carefully characterize the solution path of the iteratively reweighted $\ell_1$-penalized retire estimation, adapted from the local linear approximation algorithm for folded-concave regularization. Under a mild minimum signal strength condition, we show that after as many as $\log(\log d)$ iterations the final iterate enjoys the oracle convergence rate. At each iteration, the weighted $\ell_1$-penalized convex program can be efficiently solved by a semismooth Newton coordinate descent algorithm. Numerical studies demonstrate the competitive performance of the proposed procedure compared with either non-robust or quantile regression based alternatives.