Minimum MS. E. Gerber's Lemma

TL;DR

This paper introduces a new lower bound on the output entropy of binary symmetric channels using MMSE prediction cost, often surpassing Mrs. Gerber's Lemma, with applications to binary hidden Markov processes.

Contribution

It proposes a novel entropy lower bound based on MMSE prediction, providing tighter estimates than existing lemmas for certain processes.

Findings

The MMSE-based bound is often tighter than Mrs. Gerber's Lemma.

Application to binary hidden Markov processes yields improved entropy rate estimates.

The new bound enhances understanding of input-output entropy relationships in binary channels.

Abstract

Mrs. Gerber's Lemma lower bounds the entropy at the output of a binary symmetric channel in terms of the entropy of the input process. In this paper, we lower bound the output entropy via a different measure of input uncertainty, pertaining to the minimum mean squared error (MMSE) prediction cost of the input process. We show that in many cases our bound is tighter than the one obtained from Mrs. Gerber's Lemma. As an application, we evaluate the bound for binary hidden Markov processes, and obtain new estimates for the entropy rate.