Directed Information on Abstract Spaces: Properties and Variational Equalities

TL;DR

This paper extends the fundamental properties of directed information to abstract spaces, establishing convexity, compactness, semicontinuity, and variational equalities, which are crucial for analyzing channels with memory and feedback.

Contribution

It introduces new functional and topological properties of directed information on abstract spaces, including variational equalities analogous to mutual information.

Findings

Convexity of consistent distributions set

Weak compactness of distributions and marginals

Lower semicontinuity and continuity of directed information

Abstract

Directed information or its variants are utilized extensively in the characterization of the capacity of channels with memory and feedback, nonanticipative lossy data compression, and their generalizations to networks. In this paper, we derive several functional and topological properties of directed information for general abstract alphabets (complete separable metric spaces) using the topology of weak convergence of probability measures. These include convexity of the set of consistent distributions, which uniquely define causally conditioned distributions, convexity and concavity of directed information with respect to the sets of consistent distributions, weak compactness of these sets of distributions, their joint distributions and their marginals. Furthermore, we show lower semicontinuity of directed information, and under certain conditions we also establish continuity of directed information. Finally, we derive variational equalities for directed information, including sequential versions. These may be viewed as the analogue of the variational equalities of mutual information (utilized in Blahut-Arimoto algorithm). In summary, we extend the basic functional and topological properties of mutual information to directed information. These properties are discussed in the context of extremum problems of directed information.