On the Relationship between Mutual Information and Minimum Mean-Square Errors in Stochastic Dynamical Systems

TL;DR

This paper explores the fundamental links between mutual information and minimum mean-square errors in general stochastic dynamical systems, extending classical results from communication theory to a broader class of engineering systems.

Contribution

It establishes new relationships between mutual information and MMSE in stochastic systems described by Itô's SDEs, including both causal and non-causal errors, generalizing known Gaussian channel results.

Findings

Derived time-averaged mutual information-MMSE relationships.

Presented instantaneous, dynamical counterparts of these relationships.

Identified conditions for meaningful signal-to-noise ratio interpretation.

Abstract

We consider a general stochastic input-output dynamical system with output evolving in time as the solution to a functional coefficients, It\^{o}'s stochastic differential equation, excited by an input process. This general class of stochastic systems encompasses not only the classical communication channel models, but also a wide variety of engineering systems appearing through a whole range of applications. For this general setting we find analogous of known relationships linking input-output mutual information and minimum mean causal and non-causal square errors, previously established in the context of additive Gaussian noise communication channels. Relationships are not only established in terms of time-averaged quantities, but also their time-instantaneous, dynamical counterparts are presented. The problem of appropriately introducing in this general framework a signal-to-noise ratio notion expressed through a signal-to-noise ratio parameter is also taken into account, identifying conditions for a proper and meaningful interpretation.