Minimum Redundancy Coding for Uncertain Sources

TL;DR

This paper explores robust lossless source coding strategies within a relative entropy uncertainty set, extending traditional redundancy minimization to more practical coding schemes and analyzing their theoretical properties.

Contribution

It introduces new formulations for minimax redundancy and maximal pointwise redundancy under source uncertainty, linking them to exponential Huffman coding and practical prefix codes.

Findings

Extended exponential Huffman coding to uncertainty sets

Analyzed equivalence between minimax redundancy and Gawrychowski-Gagie problem

Provided insights into robust coding under source distribution ambiguity

Abstract

Consider the set of source distributions within a fixed maximum relative entropy with respect to a given nominal distribution. Lossless source coding over this relative entropy ball can be approached in more than one way. A problem previously considered is finding a minimax average length source code. The minimizing players are the codeword lengths --- real numbers for arithmetic codes, integers for prefix codes --- while the maximizing players are the uncertain source distributions. Another traditional minimizing objective is the first one considered here, maximum (average) redundancy. This problem reduces to an extension of an exponential Huffman objective treated in the literature but heretofore without direct practical application. In addition to these, this paper examines the related problem of maximal minimax pointwise redundancy and the problem considered by Gawrychowski and Gagie, which, for a sufficiently small relative entropy ball, is equivalent to minimax redundancy. One can consider both Shannon-like coding based on optimal real number ("ideal") codeword lengths and a Huffman-like optimal prefix coding.