Bounds for Algorithmic Mutual Information and a Unifilar Order Estimator

TL;DR

This paper explores the relationship between the growth of algorithmic mutual information and unifilar order estimators in stationary ergodic sources, providing theoretical bounds and examples relevant to natural language modeling.

Contribution

It introduces a novel, intractable order estimator with proven consistency and links it to the growth of algorithmic mutual information, extending understanding of unifilar sources.

Findings

Finite unifilar order sources exhibit sub-power-law growth of mutual information.

Infinite unifilar order sources can show power-law growth of mutual information.

Theoretical bounds connect mutual information growth to unifilar order estimates.

Abstract

Inspired by Hilberg's hypothesis, which states that mutual information between blocks for natural language grows like a power law, we seek for links between power-law growth rate of algorithmic mutual information and of some estimator of the unifilar order, i.e., the number of hidden states in the generating stationary ergodic source in its minimal unifilar hidden Markov representation. We consider an order estimator which returns the smallest order for which the maximum likelihood is larger than a weakly penalized universal probability. This order estimator is intractable and follows the ideas by Merhav, Gutman, and Ziv (1989) and by Ziv and Merhav (1992) but in its exact form seems overlooked despite some nice theoretical properties. In particular, we can prove both strong consistency of this order estimator and an upper bound of algorithmic mutual information in terms of it. Using both results, we show that all (also uncomputable) sources of a finite unifilar order exhibit sub-power-law growth of algorithmic mutual information and of the unifilar order estimator. In contrast, we also exhibit an example of unifilar processes of a countably infinite order, with a deterministic pushdown automaton and an algorithmically random oracle, for which the mentioned two quantities grow as a power law with the same exponent. We also relate our results to natural language research.