Word-Valued Sources: an Ergodic Theorem, an AEP and the Conservation of Entropy

TL;DR

This paper establishes conditions under which word-valued sources derived from a process and codebook satisfy an ergodic theorem and AEP, linking entropy rates of the source and coded process.

Contribution

It proves that asymptotic mean stationarity of the original process ensures ergodic properties and AEP for the coded source, and shows entropy rate preservation with prefix-free codes.

Findings

Word-valued sources satisfy an ergodic theorem under asymptotic mean stationarity.

These sources possess an AEP under the same conditions.

Entropy rate of the coded source equals the original entropy rate divided by average codeword length.

Abstract

A word-valued source $\mathbf{Y} = Y_1,Y_2,...$ is discrete random process that is formed by sequentially encoding the symbols of a random process $\mathbf{X} = X_1,X_2,...$ with codewords from a codebook $\mathscr{C}$. These processes appear frequently in information theory (in particular, in the analysis of source-coding algorithms), so it is of interest to give conditions on $\mathbf{X}$ and $\mathscr{C}$ for which $\mathbf{Y}$ will satisfy an ergodic theorem and possess an Asymptotic Equipartition Property (AEP). In this correspondence, we prove the following: (1) if $\mathbf{X}$ is asymptotically mean stationary, then $\mathbf{Y}$ will satisfy a pointwise ergodic theorem and possess an AEP; and, (2) if the codebook $\mathscr{C}$ is prefix-free, then the entropy rate of $\mathbf{Y}$ is equal to the entropy rate of $\mathbf{X}$ normalized by the average codeword length.