On the Information Loss in Memoryless Systems: The Multivariate Case

TL;DR

This paper defines and analyzes information loss in static, memoryless systems with continuous inputs, providing quantification, bounds, and identifying systems with infinite loss, such as quantizers and limiters.

Contribution

It introduces a concise system-theoretic definition of information loss and quantifies it for certain multivariate systems, including bounds and classes with infinite loss.

Findings

Information loss can be quantified for specific multivariate systems.

Upper bounds for information loss are derived and easy to evaluate.

Quantizers and limiters necessarily cause infinite information loss.

Abstract

In this work we give a concise definition of information loss from a system-theoretic point of view. Based on this definition, we analyze the information loss in static input-output systems subject to a continuous-valued input. For a certain class of multiple-input, multiple-output systems the information loss is quantified. An interpretation of this loss is accompanied by upper bounds which are simple to evaluate. Finally, a class of systems is identified for which the information loss is necessarily infinite. Quantizers and limiters are shown to belong to this class.