Extrinsic Jensen-Shannon Divergence: Applications to Variable-Length Coding

TL;DR

This paper introduces the extrinsic Jensen-Shannon divergence as a new tool to analyze variable-length coding over DMCs with noiseless feedback, providing bounds and schemes that achieve capacity and optimal error exponents.

Contribution

It proposes the EJS divergence for non-asymptotic analysis of variable-length coding schemes and develops new coding strategies with guaranteed rate and error performance.

Findings

EJS divergence provides non-asymptotic bounds on expected code length.

New coding schemes achieve capacity and optimal error exponents.

Simpler schemes are proposed for symmetric binary-input channels.

Abstract

This paper considers the problem of variable-length coding over a discrete memoryless channel (DMC) with noiseless feedback. The paper provides a stochastic control view of the problem whose solution is analyzed via a newly proposed symmetrized divergence, termed extrinsic Jensen-Shannon (EJS) divergence. It is shown that strictly positive lower bounds on EJS divergence provide non-asymptotic upper bounds on the expected code length. The paper presents strictly positive lower bounds on EJS divergence, and hence non-asymptotic upper bounds on the expected code length, for the following two coding schemes: variable-length posterior matching and MaxEJS coding scheme which is based on a greedy maximization of the EJS divergence. As an asymptotic corollary of the main results, this paper also provides a rate-reliability test. Variable-length coding schemes that satisfy the condition(s) of the test for parameters $R$ and $E$, are guaranteed to achieve rate $R$ and error exponent $E$. The results are specialized for posterior matching and MaxEJS to obtain deterministic one-phase coding schemes achieving capacity and optimal error exponent. For the special case of symmetric binary-input channels, simpler deterministic schemes of optimal performance are proposed and analyzed.