Relative entropy via non-sequential recursive pair substitutions

TL;DR

This paper demonstrates that the entropy, cross entropy, and Kullback-Leibler divergence of ergodic sources can be derived through successive non-sequential recursive pair substitutions, providing a new perspective on information measures.

Contribution

It extends previous work by showing that cross entropy and KL divergence can also be obtained via recursive pair substitutions, not just entropy.

Findings

Entropy can be derived using recursive pair substitutions.

Cross entropy and KL divergence are obtainable through the same method.

Provides a new approach to calculating information-theoretic measures.

Abstract

The entropy of an ergodic source is the limit of properly rescaled 1-block entropies of sources obtained applying successive non-sequential recursive pairs substitutions (see P. Grassberger 2002 ArXiv:physics/0207023 and D. Benedetto, E. Caglioti and D. Gabrielli 2006 Jour. Stat. Mech. Theo. Exp. 09 doi:10.1088/1742.-5468/2006/09/P09011). In this paper we prove that the cross entropy and the Kullback-Leibler divergence can be obtained in a similar way.