Estimation for High-Dimensional Multi-Layer Generalized Linear Model -- Part I: The Exact MMSE Estimator

TL;DR

This paper analyzes the MMSE estimation problem for high-dimensional multi-layer generalized linear models, establishing a decoupling principle and proposing an efficient approximate estimator with optimal asymptotic performance.

Contribution

It introduces a novel decoupling principle for ML-GLM estimation and proposes a low-complexity approximate estimator with asymptotic optimality.

Findings

Decoupling principle reduces complex MIMO estimation to a simple AWGN problem.

Explicit analytical equations determine the AWGN variance based on model parameters.

ML-GAMP achieves asymptotic MSE comparable to the exact MMSE estimator.

Abstract

This two-part work considers the minimum means square error (MMSE) estimation problem for a high dimensional multi-layer generalized linear model (ML-GLM), which resembles a feed-forward fully connected deep learning network in that each of its layer mixes up the random input with a known weighting matrix and activates the results via non-linear functions, except that the activation here is stochastic and following some random distribution. Part I of the work focuses on the exact MMSE estimator, whose implementation is long known infeasible. For this exact estimator, an asymptotic analysis on the performance is carried out using a new replica method that is refined from certain aspects. A decoupling principle is then established, suggesting that, in terms of joint input-and-estimate distribution, the original estimation problem of multiple-input multiple-output is indeed identical to a simple single-input single-output one subjected to additive white Gaussian noise (AWGN) only. The variance of the AWGN is further shown to be determined by some coupled equations, whose dependency on the weighting and activation is given explicitly and analytically. Comparing to existing results, this paper is the first to offer a decoupling principle for the ML-GLM estimation problem. To further address the implementation issue of an exact solution, Part II proposes an approximate estimator, ML-GAMP, whose per-iteration complexity is as low as GAMP, while its asymptotic MSE (if converged) is as optimal as the exact MMSE estimator.