Pointwise Relations between Information and Estimation in Gaussian Noise

TL;DR

This paper investigates pointwise relationships between information and estimation in Gaussian noise, providing explicit characterizations of random quantities that underpin classical expectation identities, thereby offering deeper insights into their probabilistic structure.

Contribution

It introduces pointwise relations between information and estimation in Gaussian noise, extending classical identities to the level of individual random variables rather than expectations.

Findings

Explicit characterization of the random variable difference in Duncan's mutual information and causal MMSE.

Variance of the identified random variable equals the causal MMSE and mutual information.

Provides new probabilistic insights into classical information-estimation identities.

Abstract

Many of the classical and recent relations between information and estimation in the presence of Gaussian noise can be viewed as identities between expectations of random quantities. These include the I-MMSE relationship of Guo et al.; the relative entropy and mismatched estimation relationship of Verd\'{u}; the relationship between causal estimation and mutual information of Duncan, and its extension to the presence of feedback by Kadota et al.; the relationship between causal and non-casual estimation of Guo et al., and its mismatched version of Weissman. We dispense with the expectations and explore the nature of the pointwise relations between the respective random quantities. The pointwise relations that we find are as succinctly stated as - and give considerable insight into - the original expectation identities. As an illustration of our results, consider Duncan's 1970 discovery that the mutual information is equal to the causal MMSE in the AWGN channel, which can equivalently be expressed saying that the difference between the input-output information density and half the causal estimation error is a zero mean random variable (regardless of the distribution of the channel input). We characterize this random variable explicitly, rather than merely its expectation. Classical estimation and information theoretic quantities emerge with new and surprising roles. For example, the variance of this random variable turns out to be given by the causal MMSE (which, in turn, is equal to the mutual information by Duncan's result).