Consistency of the Plug-In Estimator of the Entropy Rate for Ergodic Processes

TL;DR

This paper investigates the conditions under which the plug-in estimator can consistently estimate the entropy rate of stationary ergodic processes, revealing sample size requirements and providing new bounds related to ergodic components.

Contribution

It establishes sample size thresholds for consistent entropy rate estimation using the plug-in estimator in ergodic processes and introduces bounds on block entropy via ergodic components.

Findings

Plug-in estimator is consistent if sample length exceeds 2^{k(h+ε)}.

Inconsistent if sample length is shorter than 2^{k(h-ε)}.

Block entropy of a process is bounded by a nonlinear function of ergodic components' average entropy.

Abstract

A plug-in estimator of entropy is the entropy of the distribution where probabilities of symbols or blocks have been replaced with their relative frequencies in the sample. Consistency and asymptotic unbiasedness of the plug-in estimator can be easily demonstrated in the IID case. In this paper, we ask whether the plug-in estimator can be used for consistent estimation of the entropy rate $h$ of a stationary ergodic process. The answer is positive if, to estimate block entropy of order $k$, we use a sample longer than $2^{k(h+\epsilon)}$, whereas it is negative if we use a sample shorter than $2^{k(h-\epsilon)}$. In particular, if we do not know the entropy rate $h$, it is sufficient to use a sample of length $(|X|+\epsilon)^{k}$ where $|X|$ is the alphabet size. The result is derived using $k$-block coding. As a by-product of our technique, we also show that the block entropy of a stationary process is bounded above by a nonlinear function of the average block entropy of its ergodic components. This inequality can be used for an alternative proof of the known fact that the entropy rate a stationary process equals the average entropy rate of its ergodic components.