Random linear estimation with rotationally-invariant designs: Asymptotics at high temperature

TL;DR

This paper proves conjectures related to the asymptotic behavior of Bayesian linear estimation with rotationally-invariant designs, under high-temperature conditions, using a second-moment method and AMP algorithms.

Contribution

It establishes the validity of conjectured formulas for mutual information, MMSE, and TAP equations in high-dimensional Bayesian linear models with rotationally-invariant matrices.

Findings

Proved conjectures for mutual information and MMSE in high-dimensional limits.

Validated TAP mean-field equations for a broad class of priors.

Applied a conditional second-moment method with AMP iterates for the proof.

Abstract

We study estimation in the linear model $y=A\beta^\star+\epsilon$, in a Bayesian setting where $\beta^\star$ has an entrywise i.i.d. prior and the design $A$ is rotationally-invariant in law. In the large system limit as dimension and sample size increase proportionally, a set of related conjectures have been postulated for the asymptotic mutual information, Bayes-optimal mean squared error, and TAP mean-field equations that characterize the Bayes posterior mean of $\beta^\star$. In this work, we prove these conjectures for a general class of signal priors and for arbitrary rotationally-invariant designs $A$, under a "high-temperature" condition that restricts the range of eigenvalues of $A^\top A$. Our proof uses a conditional second-moment method argument, where we condition on the iterates of a version of the Vector AMP algorithm for solving the TAP mean-field equations.