Stability of the Shannon--McMillan--Breiman Theorem under Sublinear Parsings

TL;DR

This paper proves a stability property of the Shannon-McMillan-Breiman theorem for sublinear data parsings, showing convergence of normalized log-likelihoods to entropy rate under broad conditions.

Contribution

It establishes the first stability result for the theorem under sublinear, data-dependent parsings, including robustness to perturbations and a sharp threshold for sublinearity.

Findings

Normalized sum of negative log-likelihoods converges to entropy rate

Stability persists under subextensive perturbations

Sublinearity of block count is the sharp threshold for validity

Abstract

We establish a stability result for the Shannon-McMillan-Breiman theorem on the one-sided finite shift space. For any shift-invariant probability measure P and any data-dependent parsing whose number of blocks is sublinear in N almost surely, we show that the normalized sum of the negative log-likelihoods of the parsing blocks converges almost surely and in L^1(P) to the entropy-rate function h_P. Equivalently, we obtain an approximate factorization of cylinder probabilities under arbitrary sublinear parsings. We further show that the stability result persists under subextensive perturbations of the parsing blocks, and that sublinearity of the block count is the sharp threshold for validity at this level of generality, via a direct counterexample.