On Unique Decodability

TL;DR

This paper revisits the fundamental theorems of lossless coding, clarifies the conditions under which the inequality relating expected code length and entropy holds, especially for sources with memory, and extends McMillan's theorem for Markov sources.

Contribution

It clarifies the conditions for unique decodability inequalities and extends McMillan's theorem to Markovian sources, addressing misconceptions for sources with memory.

Findings

The inequality E[l(X_1...X_n)] >= H(X_1...X_n) holds under specific conditions for sources with memory.

McMillan's theorem is extended to apply to Markovian sources.

The paper clarifies the applicability of lossless coding theorems to practical codes and sources with memory.

Abstract

In this paper we propose a revisitation of the topic of unique decodability and of some fundamental theorems of lossless coding. It is widely believed that, for any discrete source X, every "uniquely decodable" block code satisfies E[l(X_1 X_2 ... X_n)]>= H(X_1,X_2,...,X_n), where X_1, X_2,...,X_n are the first n symbols of the source, E[l(X_1 X_2 ... X_n)] is the expected length of the code for those symbols and H(X_1,X_2,...,X_n) is their joint entropy. We show that, for certain sources with memory, the above inequality only holds when a limiting definition of "uniquely decodable code" is considered. In particular, the above inequality is usually assumed to hold for any "practical code" due to a debatable application of McMillan's theorem to sources with memory. We thus propose a clarification of the topic, also providing an extended version of McMillan's theorem to be used for Markovian sources.