Universality of Approximate Message Passing with Semi-Random Matrices

TL;DR

This paper rigorously demonstrates the universality of Approximate Message Passing algorithms' behavior across a broad class of semi-random matrices, extending beyond traditional assumptions of rotational invariance.

Contribution

It proves the asymptotic universality of memory-free AMP algorithms on semi-random matrices, including those with limited randomness and deterministic structures.

Findings

AMP behavior is universal for semi-random matrices.

Standard rotational invariant ensemble is a special case.

Includes matrices with limited randomness like the signed Sine model.

Abstract

Approximate Message Passing (AMP) is a class of iterative algorithms that have found applications in many problems in high-dimensional statistics and machine learning. In its general form, AMP can be formulated as an iterative procedure driven by a matrix $\mathbf{M}$. Theoretical analyses of AMP typically assume strong distributional properties on $\mathbf{M}$ such as $\mathbf{M}$ has i.i.d. sub-Gaussian entries or is drawn from a rotational invariant ensemble. However, numerical experiments suggest that the behavior of AMP is universal, as long as the eigenvectors of $\mathbf{M}$ are generic. In this paper, we take the first step in rigorously understanding this universality phenomenon. In particular, we investigate a class of memory-free AMP algorithms (proposed by \c{C}akmak and Opper for mean-field Ising spin glasses), and show that their asymptotic dynamics is universal on a broad class of semi-random matrices. In addition to having the standard rotational invariant ensemble as a special case, the class of semi-random matrices that we define in this work also includes matrices constructed with very limited randomness. One such example is a randomly signed version of the Sine model, introduced by Marinari, Parisi, Potters, and Ritort for spin glasses with fully deterministic couplings.