Delay and Redundancy in Lossless Source Coding

TL;DR

This paper explores how imposing delay constraints affects redundancy in lossless source coding, revealing that sequential codes can achieve exponentially decaying redundancy, unlike block codes which decay polynomially.

Contribution

It demonstrates that sequential codes can attain exponential redundancy decay with delay, and establishes bounds related to Rénnyi entropy and source symbol probabilities.

Findings

Sequential codes achieve exponential redundancy decay.

Polynomial decay is typical for block-to-variable codes.

Redundancy-delay exponent relates to Rénnyi entropy and source probabilities.

Abstract

The penalty incurred by imposing a finite delay constraint in lossless source coding of a memoryless source is investigated. It is well known that for the so-called block-to-variable and variable-to-variable codes, the redundancy decays at best polynomially with the delay, where in this case the delay is identified with the source block length or maximal source phrase length, respectively. In stark contrast, it is shown that for sequential codes (e.g., a delay-limited arithmetic code) the redundancy can be made to decay exponentially with the delay constraint. The corresponding redundancy-delay exponent is shown to be at least as good as the R\'enyi entropy of order 2 of the source, but (for almost all sources) not better than a quantity depending on the minimal source symbol probability and the alphabet size.