Exact computation of Transfer Entropy with Path Weight Sampling

TL;DR

This paper introduces TE-PWS, a novel algorithm that exactly computes transfer entropy in complex stochastic systems, overcoming limitations of previous approximation methods and revealing new insights into information flow with feedback.

Contribution

The paper presents TE-PWS, the first exact computational method for transfer entropy applicable to any stochastic model, including those with hidden variables and nonlinearity.

Findings

Approximate transfer entropy methods have large systematic errors.

TE-PWS efficiently computes transfer entropy via Monte-Carlo sampling.

Transfer entropy can reveal information flow in feedback systems beyond naive data processing.

Abstract

The ability to quantify the directional flow of information is vital to understanding natural systems and designing engineered information-processing systems. A widely used measure to quantify this information flow is the transfer entropy. However, until now, this quantity could only be obtained in dynamical models using approximations that are typically uncontrolled. Here we introduce a computational algorithm called Transfer Entropy-Path Weight Sampling (TE-PWS), which makes it possible, for the first time, to quantify the transfer entropy and its variants exactly for any stochastic model, including those with multiple hidden variables, nonlinearity, transient conditions, and feedback. By leveraging techniques from polymer and path sampling, TE-PWS efficiently computes the transfer entropy as a Monte-Carlo average over signal trajectory space. We use our exact technique to demonstrate that commonly used approximate methods to compute transfer entropies incur large systematic errors and high computational costs. As an application, we use TE-PWS in linear and nonlinear systems to reveal how transfer entropy can overcome naive applications of the data processing inequality in the presence of feedback.