Heavy Lasso: sparse penalized regression under heavy-tailed noise via data-augmented soft-thresholding

TL;DR

Heavy Lasso is a robust high-dimensional regression method that uses a Student's t-inspired loss with Lasso penalty, effectively handling heavy-tailed noise and outliers with theoretical guarantees and efficient algorithms.

Contribution

It introduces a novel loss function combining robustness to heavy-tailed noise with Lasso regularization, supported by theoretical bounds and practical algorithms.

Findings

Outperforms classical Lasso in heavy-tailed noise scenarios

Achieves comparable rates to Huber loss with robust properties

Demonstrates superior empirical performance in simulations

Abstract

High-dimensional linear regression is a fundamental tool in modern statistics, particularly when the number of predictors exceeds the sample size. The classical Lasso, which relies on the squared loss, performs well under Gaussian noise assumptions but often deteriorates in the presence of heavy-tailed errors or outliers commonly encountered in real data applications such as genomics, finance, and signal processing. To address these challenges, we propose a novel robust regression method, termed Heavy Lasso, which incorporates a loss function inspired by the Student's t-distribution within a Lasso penalization framework. This loss retains the desirable quadratic behavior for small residuals while adaptively downweighting large deviations, thus enhancing robustness to heavy-tailed noise and outliers. Heavy Lasso enjoys computationally efficient by leveraging a data augmentation scheme and a soft-thresholding algorithm, which integrate seamlessly with classical Lasso solvers. Theoretically, we establish non-asymptotic bounds under both $\ell_1$ and $\ell_2 $ norms, by employing the framework of localized convexity, showing that the Heavy Lasso estimator achieves rates comparable to those of the Huber loss. Extensive numerical studies demonstrate Heavy Lasso's superior performance over classical Lasso and other robust variants, highlighting its effectiveness in challenging noisy settings. Our method is implemented in the R package heavylasso available on Github.