On Hidden Markov Processes with Infinite Excess Entropy

TL;DR

This paper explores stationary hidden Markov processes with infinite mutual information between past and future, showing how the mutual information scales with block length and providing examples with different ergodic properties.

Contribution

It demonstrates that the block mutual information in such processes is bounded by a power law related to the hidden state distribution's tail index, and provides explicit examples.

Findings

Mutual information can grow unboundedly in these processes.

The mutual information is upper bounded by a power law depending on the tail index.

Examples include nonergodic and ergodic processes with different mutual information behaviors.

Abstract

We investigate stationary hidden Markov processes for which mutual information between the past and the future is infinite. It is assumed that the number of observable states is finite and the number of hidden states is countably infinite. Under this assumption, we show that the block mutual information of a hidden Markov process is upper bounded by a power law determined by the tail index of the hidden state distribution. Moreover, we exhibit three examples of processes. The first example, considered previously, is nonergodic and the mutual information between the blocks is bounded by the logarithm of the block length. The second example is also nonergodic but the mutual information between the blocks obeys a power law. The third example obeys the power law and is ergodic.