Optimal Estimation in Orthogonally Invariant Generalized Linear Models: Spectral Initialization and Approximate Message Passing

TL;DR

This paper develops optimal spectral estimators and an AMP algorithm for parameter estimation in orthogonally invariant generalized linear models, providing rigorous guarantees and demonstrating effectiveness beyond idealized assumptions.

Contribution

It introduces a new framework combining spectral initialization with AMP for complex correlated data, extending theoretical understanding and practical algorithms.

Findings

Spectral initialization achieves optimal sample complexity for weak recovery.

AMP algorithm attains the best possible estimation errors.

Methods outperform traditional i.i.d. assumptions in numerical tests.

Abstract

We consider the problem of parameter estimation from a generalized linear model with a random design matrix that is orthogonally invariant in law. Such a model allows the design have an arbitrary distribution of singular values and only assumes that its singular vectors are generic. It is a vast generalization of the i.i.d. Gaussian design typically considered in the theoretical literature, and is motivated by the fact that real data often have a complex correlation structure so that methods relying on i.i.d. assumptions can be highly suboptimal. Building on the paradigm of spectrally-initialized iterative optimization, this paper proposes optimal spectral estimators and combines them with an approximate message passing (AMP) algorithm, establishing rigorous performance guarantees for these two algorithmic steps. Both the spectral initialization and the subsequent AMP meet existing conjectures on the fundamental limits to estimation -- the former on the optimal sample complexity for efficient weak recovery, and the latter on the optimal errors. Numerical experiments suggest the effectiveness of our methods and accuracy of our theory beyond orthogonally invariant data.