Optimum estimation via gradients of partition functions and information measures: a statistical-mechanical perspective

TL;DR

This paper introduces a new statistical-mechanical framework linking optimal estimation to information measures like the Fisher information, applicable to a broad range of sources and channels, and demonstrates its effectiveness through examples and generalizations.

Contribution

It proposes a direct relationship between estimation and information measures viewed as partition functions, extending analysis beyond Gaussian channels to general sources and channels.

Findings

Derivation of the conditional mean estimator and MMSE using the new approach

Extension of the method to mismatched estimation scenarios

Approximate estimators with single-letter flavor for broad source/channel classes

Abstract

In continuation to a recent work on the statistical--mechanical analysis of minimum mean square error (MMSE) estimation in Gaussian noise via its relation to the mutual information (the I-MMSE relation), here we propose a simple and more direct relationship between optimum estimation and certain information measures (e.g., the information density and the Fisher information), which can be viewed as partition functions and hence are amenable to analysis using statistical--mechanical techniques. The proposed approach has several advantages, most notably, its applicability to general sources and channels, as opposed to the I-MMSE relation and its variants which hold only for certain classes of channels (e.g., additive white Gaussian noise channels). We then demonstrate the derivation of the conditional mean estimator and the MMSE in a few examples. Two of these examples turn out to be generalizable to a fairly wide class of sources and channels. For this class, the proposed approach is shown to yield an approximate conditional mean estimator and an MMSE formula that has the flavor of a single-letter expression. We also show how our approach can easily be generalized to situations of mismatched estimation.