Divergent Predictive States: The Statistical Complexity Dimension of Stationary, Ergodic Hidden Markov Processes

TL;DR

This paper introduces a method to measure the structural complexity of stationary, ergodic hidden Markov processes by calculating their statistical complexity dimension, which quantifies the divergence rate of predictive resources needed.

Contribution

It extends previous work by providing a way to determine the complexity dimension of hidden Markov processes, revealing their intrinsic structural complexity.

Findings

Calculated the Shannon entropy rate for complex processes

Constructed minimal predictive models with infinite states

Quantified the divergence rate of predictive resources

Abstract

Even simply-defined, finite-state generators produce stochastic processes that require tracking an uncountable infinity of probabilistic features for optimal prediction. For processes generated by hidden Markov chains the consequences are dramatic. Their predictive models are generically infinite-state. And, until recently, one could determine neither their intrinsic randomness nor structural complexity. The prequel, though, introduced methods to accurately calculate the Shannon entropy rate (randomness) and to constructively determine their minimal (though, infinite) set of predictive features. Leveraging this, we address the complementary challenge of determining how structured hidden Markov processes are by calculating their statistical complexity dimension -- the information dimension of the minimal set of predictive features. This tracks the divergence rate of the minimal memory resources required to optimally predict a broad class of truly complex processes.