On analytic properties of entropy rate

TL;DR

This paper investigates the analytic properties of entropy rate for discrete sources, proving Lipschitz continuity relative to total variation topology and establishing the existence of entropy rate for sources with finite evolution dimension, including hidden Markov sources and quantum walks.

Contribution

It demonstrates that entropy rate is Lipschitz continuous in total variation topology and provides a simple proof for the existence of entropy rate in sources with finite evolution dimension.

Findings

Entropy rate is Lipschitzian relative to total variation topology.

Existence of entropy rate is proven for sources with finite evolution dimension.

Includes sources like hidden Markov models and quantum random walks.

Abstract

Entropy rate is a real valued functional on the space of discrete random sources which lacks a closed formula even for subclasses of sources which have intuitive parameterizations. A good way to overcome this problem is to examine its analytic properties relative to some reasonable topology. A canonical choice of a topology is that of the norm of total variation as it immediately arises with the idea of a discrete random source as a probability measure on sequence space. It is shown that entropy rate is Lipschitzian relative to this topology, which, by well known facts, is close to differentiability. An application of this theorem leads to a simple and elementary proof of the existence of entropy rate of random sources with finite evolution dimension. This class of sources encompasses arbitrary hidden Markov sources and quantum random walks.