On Data-Processing and Majorization Inequalities for $f$-Divergences with Applications

TL;DR

This paper develops data-processing and majorization inequalities for $f$-divergences, applying them to list decoding performance, probability approximation, and Tunstall code compression, with extensive analytical and numerical results.

Contribution

It introduces new inequalities for $f$-divergences and applies them to improve bounds in list decoding and analyze Tunstall code performance.

Findings

Unified bounds on list decoding error probability.

Numerical bounds for list decoding performance.

Analysis of Tunstall code compression rates.

Abstract

This paper is focused on derivations of data-processing and majorization inequalities for $f$-divergences, and their applications in information theory and statistics. For the accessibility of the material, the main results are first introduced without proofs, followed by exemplifications of the theorems with further related analytical results, interpretations, and information-theoretic applications. One application refers to the performance analysis of list decoding with either fixed or variable list sizes; some earlier bounds on the list decoding error probability are reproduced in a unified way, and new bounds are obtained and exemplified numerically. Another application is related to a study of the quality of approximating a probability mass function, induced by the leaves of a Tunstall tree, by an equiprobable distribution. The compression rates of finite-length Tunstall codes are further analyzed for asserting their closeness to the Shannon entropy of a memoryless and stationary discrete source. Almost all the analysis is relegated to the appendices, which form a major part of this manuscript.