An Information Geometry Interpretation for Approximate Message Passing

TL;DR

This paper introduces an information geometry framework for linear regression that unifies and explains the approximate message passing algorithm, providing new insights into stochastic inference methods.

Contribution

It extends information geometry to linear regression, deriving new algorithms and establishing the equivalence between AIGA and AMP, offering a novel perspective on message passing.

Findings

A new IG framework for linear regression.

Derivation of IGA and AIGA for BPDN.

Proof of AIGA's equivalence to AMP.

Abstract

In this paper, we propose an information geometry (IG) framework to solve the standard linear regression problem. The proposed framework is an extension of the one for computing the mean of complex multivariate Gaussian distribution. By applying the proposed framework, the information geometry approach (IGA) and the approximate information geometry approach (AIGA) for basis pursuit de-noising (BPDN) in standard linear regression are derived. The framework can also be applied to other standard linear regression problems. With the transformations of natural and expectation parameters of Gaussian distributions, we then show the relationship between the IGA and the message passing (MP) algorithm. Finally, we prove that the AIGA is equivalent to the approximate message passing (AMP) algorithm. These intrinsic results offer a new perspective for the AMP algorithm, and clues for understanding and improving stochastic reasoning methods.