Transfer Entropy and Directed Information in Gaussian diffusion processes

TL;DR

This paper extends the concepts of Transfer Entropy and Directed Information to multidimensional Gaussian diffusion processes, providing new definitions, computational methods, and insights into their properties in continuous time.

Contribution

It introduces two new definitions of Directed Information for Gaussian diffusions, ensuring they capture influence accurately and are computationally tractable via Riccati equations.

Findings

Transfer Entropy and Directed Information are expressed through Riccati equations.

The paper proposes a new continuous-time Directed Information definition.

Influence measures are validated for jointly Markov components.

Abstract

Transfer Entropy and Directed Information are information-theoretic measures of the directional dependency between stochastic processes. Following the definitions of Schreiber and Massey in discrete time, we define and evaluate these measures for the components of multidimensional Gaussian diffusion processes. When the components are jointly Markov, the Transfer Entropy and Directed Information are both measures of influence according to a simple physical principle. More generally, the effect of other components has to be accounted for, and this can be achieved in more than one way. We propose two definitions, one of which preserves the properties of influence of the jointly Markov case. The Transfer Entropy and Directed Information are expressed in terms of the solutions of matrix Riccati equations, and so are easy to compute. The definition of continuous-time Directed Information we propose differs from that previously appearing in the literature. We argue that the latter is not strictly directional.