Mutual Information, Relative Entropy and Estimation Error in Semi-martingale Channels

TL;DR

This paper extends fundamental information-estimation relations from Gaussian and Poisson channels to a broader class of semi-martingale channels, unifying continuous-time models and including mismatched estimation scenarios.

Contribution

It introduces semi-martingale channels and establishes new representations linking mutual information to causal filtering loss, generalizing previous results.

Findings

Relations hold for semi-martingale channels including Gaussian and Poisson cases

Mutual information equals optimal causal filtering loss in these channels

Relative entropy corresponds to cumulative mismatched estimation loss

Abstract

Fundamental relations between information and estimation have been established in the literature for the continuous-time Gaussian and Poisson channels, in a long line of work starting from the classical representation theorems by Duncan and Kabanov respectively. In this work, we demonstrate that such relations hold for a much larger family of continuous-time channels. We introduce the family of semi-martingale channels where the channel output is a semi-martingale stochastic process, and the channel input modulates the characteristics of the semi-martingale. For these channels, which includes as a special case the continuous time Gaussian and Poisson models, we establish new representations relating the mutual information between the channel input and output to an optimal causal filtering loss, thereby unifying and considerably extending results from the Gaussian and Poisson settings. Extensions to the setting of mismatched estimation are also presented where the relative entropy between the laws governing the output of the channel under two different input distributions is equal to the cumulative difference between the estimation loss incurred by using the mismatched and optimal causal filters respectively. The main tool underlying these results is the Doob--Meyer decomposition of a class of likelihood ratio sub-martingales. The results in this work can be viewed as the continuous-time analogues of recent generalizations for relations between information and estimation for discrete-time L\'evy channels.