On the Renyi Divergence, Joint Range of Relative Entropies, and a Channel Coding Theorem

TL;DR

This paper explores the geometric and optimization properties of Renyi divergence and total variation distance, providing new bounds on channel coding performance and insights into the joint range of relative entropies.

Contribution

It characterizes the joint range of relative entropies under total variation constraints and derives a new exponential upper bound on binary linear code performance using Renyi divergence.

Findings

Exact locus of (D(Q||P1), D(Q||P2)) determined for probability measures with total variation constraints.

All points in the convex region are attained by binary alphabet measures.

Derived exponential upper bound on binary linear code performance based on Renyi divergence.

Abstract

This paper starts by considering the minimization of the Renyi divergence subject to a constraint on the total variation distance. Based on the solution of this optimization problem, the exact locus of the points $\bigl( D(Q\|P_1), D(Q\|P_2) \bigr)$ is determined when $P_1, P_2, Q$ are arbitrary probability measures which are mutually absolutely continuous, and the total variation distance between $P_1$ and $P_2$ is not below a given value. It is further shown that all the points of this convex region are attained by probability measures which are defined on a binary alphabet. This characterization yields a geometric interpretation of the minimal Chernoff information subject to a constraint on the total variation distance. This paper also derives an exponential upper bound on the performance of binary linear block codes (or code ensembles) under maximum-likelihood decoding. Its derivation relies on the Gallager bounding technique, and it reproduces the Shulman-Feder bound as a special case. The bound is expressed in terms of the Renyi divergence from the normalized distance spectrum of the code (or the average distance spectrum of the ensemble) to the binomially distributed distance spectrum of the capacity-achieving ensemble of random block codes. This exponential bound provides a quantitative measure of the degradation in performance of binary linear block codes (or code ensembles) as a function of the deviation of their distance spectra from the binomial distribution. An efficient use of this bound is considered.