Directed Information on Abstract spaces: Properties and Extremum Problems

TL;DR

This paper develops a framework for directed information on abstract spaces, establishing key properties and solving extremum problems related to channel capacity and rate distortion, with implications for information theory.

Contribution

It introduces a novel abstract space framework for directed information and analyzes extremum problems, proving existence of optimal distributions for channels with memory and feedback.

Findings

Directed information is shown to be convex, concave, and lower semicontinuous.

Existence of maximizing and minimizing distributions for extremum problems is established.

The framework applies to channels with memory, feedback, and rate distortion scenarios.

Abstract

This paper describes a framework in which directed information is defined on abstract spaces. The framework is employed to derive properties of directed information such as convexity, concavity, lower semicontinuity, by using the topology of weak convergence of probability measures on Polish spaces. Two extremum problems of directed information related to capacity of channels with memory and feedback, and non-anticipative and sequential rate distortion are analyzed showing existence of maximizing and minimizing distributions, respectively.