Equivalence of additive-combinatorial linear inequalities for Shannon entropy and differential entropy

TL;DR

This paper establishes a precise equivalence between linear inequalities involving Shannon entropy and differential entropy for sums of independent group-valued random variables, unifying discrete and continuous cases.

Contribution

It proves that balanced linear inequalities of Shannon entropy hold if and only if their differential entropy counterparts hold, extending to certain abelian groups and unifying discrete and continuous entropy inequalities.

Findings

Balanced inequalities of Shannon entropy are equivalent to those of differential entropy.

Differential entropy inequalities by Kontoyiannis and Madiman follow from Tao's discrete inequalities.

The results apply to sums of independent group-valued random variables and certain abelian groups.

Abstract

This paper addresses the correspondence between linear inequalities of Shannon entropy and differential entropy for sums of independent group-valued random variables. We show that any balanced (with the sum of coefficients being zero) linear inequality of Shannon entropy holds if and only if its differential entropy counterpart also holds; moreover, any linear inequality for differential entropy must be balanced. In particular, our result shows that recently proved differential entropy inequalities by Kontoyiannis and Madiman \cite{KM14} can be deduced from their discrete counterparts due to Tao \cite{Tao10} in a unified manner. Generalizations to certain abelian groups are also obtained. Our proof of extending inequalities of Shannon entropy to differential entropy relies on a result of R\'enyi \cite{Renyi59} which relates the Shannon entropy of a finely discretized random variable to its differential entropy and also helps in establishing the entropy of the sum of quantized random variables is asymptotically equal to that of the quantized sum; the converse uses the asymptotics of the differential entropy of convolutions with weak additive noise.