Information Measures for Deterministic Input-Output Systems

TL;DR

This paper investigates information loss in deterministic, memoryless systems by analyzing conditional entropy, establishing bounds on reconstruction error, and relating information loss to Rényi information dimension, with practical examples.

Contribution

It introduces a framework for quantifying information loss in deterministic systems and derives bounds linking loss to reconstruction error, extending information theory concepts to such systems.

Findings

Information loss is finite for many systems, even with continuous inputs.

Fano-type bounds relate information loss to reconstruction error probability.

A relative measure of information loss is connected to Rényi information dimension.

Abstract

In this work the information loss in deterministic, memoryless systems is investigated by evaluating the conditional entropy of the input random variable given the output random variable. It is shown that for a large class of systems the information loss is finite, even if the input is continuously distributed. Based on this finiteness, the problem of perfectly reconstructing the input is addressed and Fano-type bounds between the information loss and the reconstruction error probability are derived. For systems with infinite information loss a relative measure is defined and shown to be tightly related to R\'{e}nyi information dimension. Employing another Fano-type argument, the reconstruction error probability is bounded by the relative information loss from below. In view of developing a system theory from an information-theoretic point-of-view, the theoretical results are illustrated by a few example systems, among them a multi-channel autocorrelation receiver.