Signal-to-noise ratio aware minimax analysis of sparse linear regression

TL;DR

This paper introduces an SNR-aware minimax framework for sparse linear regression, revealing how the signal-to-noise ratio critically influences the optimal estimation strategies across different regimes.

Contribution

It develops a higher-order asymptotic analysis to incorporate SNR effects into minimax risk, providing new insights into sparse signal estimation under noise.

Findings

Identifies three SNR regimes with distinct estimator behaviors

Reveals limitations of existing minimax analyses ignoring SNR effects

Offers improved theoretical understanding aligning with empirical results

Abstract

We consider parameter estimation under sparse linear regression -- an extensively studied problem in high-dimensional statistics and compressed sensing. While the minimax framework has been one of the most fundamental approaches for studying statistical optimality in this problem, we identify two important issues that the existing minimax analyses face: (i) The signal-to-noise ratio appears to have no effect on the minimax optimality, while it shows a major impact in numerical simulations. (ii) Estimators such as best subset selection and Lasso are shown to be minimax optimal, yet they exhibit significantly different performances in simulations. In this paper, we tackle the two issues by employing a minimax framework that accounts for variations in the signal-to-noise ratio (SNR), termed the SNR-aware minimax framework. We adopt a delicate higher-order asymptotic analysis technique to obtain the SNR-aware minimax risk. Our theoretical findings determine three distinct SNR regimes: low-SNR, medium-SNR, and high-SNR, wherein minimax optimal estimators exhibit markedly different behaviors. The new theory not only offers much better elaborations for empirical results, but also brings new insights to the estimation of sparse signals in noisy data.