Time-series Random Process Complexity Ranking Using a Bound on Conditional Differential Entropy

TL;DR

This paper proposes a theoretically grounded, computationally feasible method for ranking time-series complexity using bounds on conditional differential entropy derived from prediction error covariances, validated through synthetic experiments.

Contribution

It extends existing bounds on conditional differential entropy by leveraging matrix inequalities, enabling practical complexity ranking of time series from prediction errors.

Findings

Bounded the conditional differential entropy using covariance matrix determinants.

Validated the ranking method on synthetic linear and bio-inspired data.

Demonstrated the approach's effectiveness in recovering known complexity orderings.

Abstract

Conditional differential entropy provides an intuitive measure for relatively ranking time-series complexity by quantifying uncertainty in future observations given past context. However, its direct computation for high-dimensional processes from unknown distributions is often intractable. This paper builds on the information theoretic prediction error bounds established by Fang et al. \cite{fang2019generic}, which demonstrate that the conditional differential entropy \textbf{$h(X_k \mid X_{k-1},...,X_{k-m})$} is upper bounded by a function of the determinant of the covariance matrix of next-step prediction errors for any next step prediction model. We add to this theoretical framework by further increasing this bound by leveraging Hadamard's inequality and the positive semi-definite property of covariance matrices. To see if these bounds can be used to rank the complexity of time series, we conducted two synthetic experiments: (1) controlled linear autoregressive processes with additive Gaussian noise, where we compare ordinary least squares prediction error entropy proxies to the true entropies of various additive noises, and (2) a complexity ranking task of bio-inspired synthetic audio data with unknown entropy, where neural network prediction errors are used to recover the known complexity ordering. This framework provides a computationally tractable method for time-series complexity ranking using prediction errors from next-step prediction models, that maintains a theoretical foundation in information theory.