All-or-Nothing Phenomena: From Single-Letter to High Dimensions

TL;DR

This paper investigates phase transitions in high-dimensional linear regression, showing that the MMSE exhibits a sharp threshold behavior and highlighting a potential gap between optimal estimation and computationally feasible algorithms.

Contribution

It establishes that the asymptotic MMSE converges to a step function under certain conditions and reveals a computational-statistical gap via the analysis of approximate message passing algorithms.

Findings

MMSE converges to a step function at a critical threshold.

Approximate message passing error also exhibits a step function with a higher threshold.

Evidence of a gap between statistical optimality and computational feasibility.

Abstract

We consider the linear regression problem of estimating a $p$-dimensional vector $\beta$ from $n$ observations $Y = X \beta + W$, where $\beta_j \stackrel{\text{i.i.d.}}{\sim} \pi$ for a real-valued distribution $\pi$ with zero mean and unit variance, $X_{ij} \stackrel{\text{i.i.d.}}{\sim} \mathcal{N}(0,1)$, and $W_i\stackrel{\text{i.i.d.}}{\sim} \mathcal{N}(0, \sigma^2)$. In the asymptotic regime where $n/p \to \delta$ and $ p/ \sigma^2 \to \mathsf{snr}$ for two fixed constants $\delta, \mathsf{snr}\in (0, \infty)$ as $p \to \infty$, the limiting (normalized) minimum mean-squared error (MMSE) has been characterized by the MMSE of an associated single-letter (additive Gaussian scalar) channel. In this paper, we show that if the MMSE function of the single-letter channel converges to a step function, then the limiting MMSE of estimating $\beta$ in the linear regression problem converges to a step function which jumps from $1$ to $0$ at a critical threshold. Moreover, we establish that the limiting mean-squared error of the (MSE-optimal) approximate message passing algorithm also converges to a step function with a larger threshold, providing evidence for the presence of a computational-statistical gap between the two thresholds.