Bounds on the Entropy of a Function of a Random Variable and their Applications

TL;DR

This paper derives tight bounds on the entropy of a function of a discrete random variable, especially when the function is not one-to-one, with implications for understanding entropy relationships in information theory.

Contribution

It provides the first tight bounds on $H(f(X))$ for non-injective functions and introduces an improved lower bound on distribution entropy based on probability ratio bounds.

Findings

Derived tight bounds on $H(f(X))$ for non-one-to-one functions.

Established an improved lower bound on entropy given probability ratio constraints.

Illustrated scenarios where these bounds are applicable.

Abstract

It is well known that the entropy $H(X)$ of a discrete random variable $X$ is always greater than or equal to the entropy $H(f(X))$ of a function $f$ of $X$, with equality if and only if $f$ is one-to-one. In this paper, we give tight bounds on $H(f(X))$ when the function $f$ is not one-to-one, and we illustrate a few scenarios where this matters. As an intermediate step towards our main result, we derive a lower bound on the entropy of a probability distribution, when only a bound on the ratio between the maximal and minimal probabilities is known. The lower bound improves on previous results in the literature, and it could find applications outside the present scenario.