Anatomy of a Bit: Information in a Time Series Observation

TL;DR

This paper analyzes the information content in multivariate time series observations using various information measures, revealing insights into the internal dynamics and shared information across time, and motivating the need for modeling internal states.

Contribution

It introduces new multivariate information measures and interprets their asymptotic behavior to understand the internal information dynamics of stochastic processes.

Findings

Information measures scale with time window length

Shared information between past and future exists beyond single observations

Explicit relationships between different information components are developed

Abstract

Appealing to several multivariate information measures---some familiar, some new here---we analyze the information embedded in discrete-valued stochastic time series. We dissect the uncertainty of a single observation to demonstrate how the measures' asymptotic behavior sheds structural and semantic light on the generating process's internal information dynamics. The measures scale with the length of time window, which captures both intensive (rates of growth) and subextensive components. We provide interpretations for the components, developing explicit relationships between them. We also identify the informational component shared between the past and the future that is not contained in a single observation. The existence of this component directly motivates the notion of a process's effective (internal) states and indicates why one must build models.