Predictive Information

TL;DR

This paper explores the concept of predictive information in data streams, linking it to complexity measures in dynamical systems and statistics, and argues that divergence in predictive information uniquely quantifies model complexity.

Contribution

It establishes a theoretical connection between predictive information divergence and complexity, proposing it as a consistent measure of system richness.

Findings

Predictive information diverges with increasing data, indicating model learning.

Divergence in predictive information measures complexity of the underlying system.

The approach unifies concepts from information theory, dynamical systems, and statistics.

Abstract

Observations on the past provide some hints about what will happen in the future, and this can be quantified using information theory. The ``predictive information'' defined in this way has connections to measures of complexity that have been proposed both in the study of dynamical systems and in mathematical statistics. In particular, the predictive information diverges when the observed data stream allows us to learn an increasingly precise model for the dynamics that generate the data, and the structure of this divergence measures the complexity of the model. We argue that divergent contributions to the predictive information provide the only measure of complexity or richness that is consistent with certain plausible requirements.