Information Loss in Static Nonlinearities

TL;DR

This paper quantifies the information loss caused by static nonlinearities in systems using conditional entropy, deriving formulas that relate input distributions and system properties, with practical bounds and illustrative examples.

Contribution

It introduces a novel method to quantify information loss in nonlinear systems using conditional entropy, providing explicit formulas and tight bounds based on system non-injectivity.

Findings

Derived an expression for information loss depending on input density and nonlinearity.

Established tight upper bounds for information loss that are easier to evaluate.

Demonstrated the application of the results through illustrative examples.

Abstract

In this work, conditional entropy is used to quantify the information loss induced by passing a continuous random variable through a memoryless nonlinear input-output system. We derive an expression for the information loss depending on the input density and the nonlinearity and show that the result is strongly related to the non-injectivity of the considered system. Tight upper bounds are presented, which can be evaluated with less difficulty than a direct evaluation of the information loss, which involves the logarithm of a sum. Application of our results is illustrated on a set of examples.