An Easily Tunable Approach to Robust and Sparse High-Dimensional Linear Regression

TL;DR

This paper introduces a robust, sparse linear regression method that adapts to unknown noise levels and outliers using an iterative Huber-based approach with median-of-means, providing sharp error bounds without prior noise or outlier knowledge.

Contribution

The proposed estimator eliminates the need for tuning parameters related to noise scale and outliers, achieving optimal error bounds under various noise conditions.

Findings

Achieves sharp non-asymptotic error bounds under sub-Gaussian and heavy-tailed noise.

Handles arbitrary outliers without prior knowledge of their number or sparsity.

Matches minimax lower bounds when the number of outliers is known.

Abstract

Sparse linear regression methods such as Lasso require a tuning parameter that depends on the noise variance, which is typically unknown and difficult to estimate in practice. In the presence of heavy-tailed noise or adversarial outliers, this problem becomes more challenging. In this paper, we propose an estimator for robust and sparse linear regression that eliminates the need for explicit prior knowledge of the noise scale. Our method builds on the Huber loss and incorporates an iterative scheme that alternates between coefficient estimation and adaptive noise calibration via median-of-means. The approach is theoretically grounded and achieves sharp non-asymptotic error bounds under both sub-Gaussian and heavy-tailed noise assumptions. Moreover, the proposed method accommodates arbitrary outlier contamination in the response without requiring prior knowledge of the number of outliers or the sparsity level. While previous robust estimators avoid tuning parameters related to the noise scale or sparsity, our procedure achieves comparable error bounds when the number of outliers is unknown, and improved bounds when it is known. In particular, the improved bounds match the known minimax lower bounds up to constant factors.