# Mutual Information Rate -- Linear Noise Approximation and Exact Computation

## TL;DR

This paper evaluates the accuracy of the Gaussian approximation for mutual information rate estimation in nonlinear and discrete systems, using Path Weight Sampling for exact computation and highlighting when more precise methods are needed.

## Contribution

It provides a quantitative assessment of the Gaussian approximation's limitations and demonstrates the effectiveness of Path Weight Sampling for exact mutual information rate calculation.

## Key findings

- Gaussian approximation often fails in nearly Gaussian systems
- Significant deviations occur in nonlinear diffusive systems
- Path Weight Sampling offers exact mutual information estimates

## Abstract

Efficient information processing is crucial for both living organisms and engineered systems. The mutual information rate, a core concept of information theory, quantifies the amount of information shared between the trajectories of input and output signals, and enables the quantification of information flow in dynamic systems. A common approach for estimating the mutual information rate is the Gaussian approximation which assumes that the input and output trajectories follow Gaussian statistics. However, this method is limited to linear systems, and its accuracy in nonlinear or discrete systems remains unclear. In this work, we assess the accuracy of the Gaussian approximation for non-Gaussian systems by leveraging Path Weight Sampling (PWS), a recent technique for exactly computing the mutual information rate. In two case studies, we examine the limitations of the Gaussian approximation. First, we focus on discrete linear systems and demonstrate that, even when the system's statistics are nearly Gaussian, the Gaussian approximation fails to accurately estimate the mutual information rate. Second, we explore a continuous diffusive system with a nonlinear transfer function, revealing significant deviations between the Gaussian approximation and the exact mutual information rate as nonlinearity increases. Our results provide a quantitative evaluation of the Gaussian approximation's performance across different stochastic models and highlight when more computationally intensive methods, such as PWS, are necessary.