Predictive Complexity and Generalized Entropy Rate of Stationary Ergodic Processes

TL;DR

This paper investigates the loss rate of predictors in stationary ergodic processes using generalized entropy, establishing new theoretical results including a version of the Shannon-McMillan-Breiman theorem for certain prediction games.

Contribution

It introduces a well-defined notion of generalized entropy for stationary ergodic distributions and proves convergence of predictive complexity to this entropy under specific conditions.

Findings

Generalized entropy is well-defined for stationary ergodic distributions.

A version of the Shannon-McMillan-Breiman theorem holds for regular prediction games.

Predictive complexity converges to generalized entropy almost everywhere.

Abstract

In the online prediction framework, we use generalized entropy of to study the loss rate of predictors when outcomes are drawn according to stationary ergodic distributions over the binary alphabet. We show that the notion of generalized entropy of a regular game \cite{KVV04} is well-defined for stationary ergodic distributions. In proving this, we obtain new game-theoretic proofs of some classical information theoretic inequalities. Using Birkhoff's ergodic theorem and convergence properties of conditional distributions, we prove that a classical Shannon-McMillan-Breiman theorem holds for a restricted class of regular games, when no computational constraints are imposed on the prediction strategies. If a game is mixable, then there is an optimal aggregating strategy which loses at most an additive constant when compared to any other lower semicomputable strategy. The loss incurred by this algorithm on an infinite sequence of outcomes is called its predictive complexity. We use our version of Shannon-McMillan-Breiman theorem to prove that when a restriced regular game has a predictive complexity, the predictive complexity converges to the generalized entropy of the game almost everywhere with respect to the stationary ergodic distribution.