Greedy Variance Estimation for the LASSO

TL;DR

This paper introduces a fast, efficient variance estimator for high-dimensional linear regression that outperforms traditional methods in speed and scalability while maintaining consistency under RIP conditions.

Contribution

It presents a novel variance estimator for LASSO that is faster than LASSO, scalable, and provably consistent under RIP assumptions.

Findings

Estimator is faster than LASSO, requiring only p matrix-vector multiplications.

It is highly parallelizable and scales well to high dimensions.

The estimator incurs only a modest bias while maintaining consistency.

Abstract

Recent results have proven the minimax optimality of LASSO and related algorithms for noisy linear regression. However, these results tend to rely on variance estimators that are inefficient or optimizations that are slower than LASSO itself. We propose an efficient estimator for the noise variance in high dimensional linear regression that is faster than LASSO, only requiring $p$ matrix-vector multiplications. We prove this estimator is consistent with a good rate of convergence, under the condition that the design matrix satisfies the Restricted Isometry Property (RIP). In practice, our estimator scales incredibly well into high dimensions, is highly parallelizable, and only incurs a modest bias.