Estimation for High-Dimensional Multi-Layer Generalized Linear Model -- Part II: The ML-GAMP Estimator

TL;DR

This paper introduces ML-GAMP, an efficient approximate estimator for multi-layer generalized linear models, with performance characterized by simple state evolution equations and proven to be Bayes-optimal in large systems.

Contribution

It proposes ML-GAMP, a low-complexity approximate estimator for multi-layer GLMs, with asymptotic performance analysis matching the optimal MMSE estimator.

Findings

ML-GAMP has per-iteration complexity similar to GAMP.

Its asymptotic MSE behavior is fully characterized by simple state evolution equations.

The fixed points of ML-GAMP's SE match those of the optimal MMSE estimator.

Abstract

This is Part II of a two-part work on the estimation for a multi-layer generalized linear model (ML-GLM) in large system limits. In Part I, we had analyzed the asymptotic performance of an exact MMSE estimator, and obtained a set of coupled equations that could characterize its MSE performance. To work around the implementation difficulty of the exact estimator, this paper continues to propose an approximate solution, ML-GAMP, which could be derived by blending a moment-matching projection into the Gaussian approximated loopy belief propagation. The ML-GAMP estimator is then shown to enjoy a great simplicity in its implementation, where its per-iteration complexity is as low as GAMP. Further analysis on its asymptotic performance also reveals that, in large system limits, its dynamical MSE behavior is fully characterized by a set of simple one-dimensional iterating equations, termed state evolution (SE). Interestingly, this SE of ML-GAMP share exactly the same fixed points with an exact MMSE estimator whose fixed points were obtained in Part I via a replica analysis. Given the Bayes-optimality of the exact implementation, this proposed estimator (if converged) is optimal in the MSE sense.