Non-equilibrium Information Envelopes and the Capacity-Delay-Error-Tradeoff of Source Coding

TL;DR

This paper introduces an envelope-based framework linking information theory and queueing theory, analyzing source coding dynamics and tradeoffs between capacity, delay, and error, especially on short time scales and with sources having memory.

Contribution

It develops a novel information envelope approach that captures non-equilibrium source coding behavior and characterizes capacity-delay-error tradeoffs without relying on classical assumptions.

Findings

Information envelopes converge to source entropy over time

Large deviations cause network delays in non-equilibrium conditions

Optimal coding approaches entropy with small error probability at high capacity

Abstract

This paper develops an envelope-based approach to establish a link between information and queueing theory. Unlike classical, equilibrium information theory, information envelopes focus on the dynamics of sources and coders, using functions of time that bound the number of bits generated. In the limit the information envelopes converge to the average behavior and recover the entropy of a source, respectively, the average codeword length of a coder. In contrast, on short time scales and for sources with memory it is shown that large deviations from known equilibrium results occur with non-negligible probability. These can cause significant network delays. Compared to well-known traffic models from queueing theory, information envelopes consider the functioning of information sources and coders, avoiding a priori assumptions, such as exponential traffic, or empirical, trace-based traffic models. Using results from the stochastic network calculus, the envelopes yield a characterization of the operating points of source coders by the triplet of capacity, delay, and error. In the limit, assuming an optimal coder the required capacity approaches the entropy with arbitrarily small probability of error if infinitely large delays are permitted. We derive a corresponding characterization of channels and prove that the model has the desirable property of additivity, that allows analyzing coders and channels separately.