Novel Lower Bounds on the Entropy Rate of Binary Hidden Markov Processes

TL;DR

This paper develops new lower bounds on the entropy rate of binary hidden Markov processes by applying a strengthened version of Mrs. Gerber's Lemma, improving existing bounds especially in noisy regimes.

Contribution

It introduces novel lower bounds on entropy rates of binary hidden Markov processes using advanced entropy inequalities, including explicit bounds for constrained Markov processes.

Findings

Improved lower bounds in very noisy regimes for symmetric processes

Explicit bounds for Markov processes with (1,∞)-RLL constraints

Enhanced understanding of entropy behavior in hidden Markov models

Abstract

Recently, Samorodnitsky proved a strengthened version of Mrs. Gerber's Lemma, where the output entropy of a binary symmetric channel is bounded in terms of the average entropy of the input projected on a random subset of coordinates. Here, this result is applied for deriving novel lower bounds on the entropy rate of binary hidden Markov processes. For symmetric underlying Markov processes, our bound improves upon the best known bound in the very noisy regime. The nonsymmetric case is also considered, and explicit bounds are derived for Markov processes that satisfy the $(1,\infty)$-RLL constraint.