Discrete Layered Entropy, Conditional Compression and a Tighter Strong Functional Representation Lemma

TL;DR

This paper introduces discrete layered entropy, a piecewise linear approximation of Shannon entropy with useful properties for information theory applications, leading to tighter bounds in the strong functional representation lemma.

Contribution

It defines discrete layered entropy with key properties, enabling improved bounds and approximations in entropy-related problems, notably refining the strong functional representation lemma.

Findings

Discrete layered entropy approximates Shannon entropy within a logarithmic gap.

Provides a new bound for the strong functional representation lemma that is within 2.8 bits of optimal.

Enhances methods for conditional encoding and entropy bounds in information theory.

Abstract

We study a quantity called discrete layered entropy, which approximates the Shannon entropy within a logarithmic gap. Compared to the Shannon entropy, the discrete layered entropy is piecewise linear, approximates the expected length of the optimal one-to-one non-prefix code, and satisfies an elegant conditioning property. These properties make it useful for approximating the Shannon entropy in linear programming and maximum entropy problems, studying the optimal length of conditional encoding, and bounding the entropy of monotonic mixture distributions. In particular, it can give a bound $I(X;Y)+\log(I(X;Y)+3.4)+1$ for the strong functional representation lemma which is optimal within $2.8$ bits, and significantly improves upon the best known bound.