Entropy and Source Coding for Integer-Dimensional Singular Random Variables

TL;DR

This paper introduces a new entropy concept for integer-dimensional singular random variables, bridging discrete and continuous cases, with applications in source coding and information bounds.

Contribution

It provides the first tractable entropy extension for integer-dimensional singular variables, including properties and applications in source coding.

Findings

Entropy definition unifies discrete and continuous cases.

Entropy transforms naturally under Lipschitz functions.

Applications include minimal codeword length and Shannon bounds.

Abstract

Entropy and differential entropy are important quantities in information theory. A tractable extension to singular random variables-which are neither discrete nor continuous-has not been available so far. Here, we present such an extension for the practically relevant class of integer-dimensional singular random variables. The proposed entropy definition contains the entropy of discrete random variables and the differential entropy of continuous random variables as special cases. We show that it transforms in a natural manner under Lipschitz functions, and that it is invariant under unitary transformations. We define joint entropy and conditional entropy for integer-dimensional singular random variables, and we show that the proposed entropy conveys useful expressions of the mutual information. As first applications of our entropy definition, we present a result on the minimal expected codeword length of quantized integer-dimensional singular sources and a Shannon lower bound for integer-dimensional singular sources.