On the Rate of Information Loss in Memoryless Systems

TL;DR

This paper investigates how much information is lost when a stationary stochastic process passes through a memoryless system, relating the loss to differential entropy rates and providing bounds and conditions for Markovian properties.

Contribution

It establishes a relationship between information loss rate and differential entropy rates for certain systems, and provides bounds and conditions for Markovian output processes.

Findings

Information loss rate relates to differential entropy rate difference.

Bounds on information loss rate are derived.

Conditions for Markovian output processes are identified.

Abstract

In this work we present results about the rate of (relative) information loss induced by passing a real-valued, stationary stochastic process through a memoryless system. We show that for a special class of systems the information loss rate is closely related to the difference of differential entropy rates of the input and output processes. It is further shown that the rate of (relative) information loss is bounded from above by the (relative) information loss the system induces on a random variable distributed according to the process's marginal distribution. As a side result, in this work we present sufficient conditions such that for a continuous-valued Markovian input process also the output process possesses the Markov property.