Circumventing the Curse of Dimensionality in Prediction: Causal Rate-Distortion for Infinite-Order Markov Processes

TL;DR

This paper introduces a causal rate-distortion framework that overcomes the curse of dimensionality in predicting infinite-order Markov processes, significantly improving analysis accuracy.

Contribution

It presents a novel causal rate-distortion approach using computational mechanics to better analyze long-range dependencies in complex stochastic processes.

Findings

Causal rate-distortion theory improves prediction of infinite-order processes.

The method handles long-range correlations effectively.

Results outperform traditional finite-sequence clustering approaches.

Abstract

Predictive rate-distortion analysis suffers from the curse of dimensionality: clustering arbitrarily long pasts to retain information about arbitrarily long futures requires resources that typically grow exponentially with length. The challenge is compounded for infinite-order Markov processes, since conditioning on finite sequences cannot capture all of their past dependencies. Spectral arguments show that algorithms which cluster finite-length sequences fail dramatically when the underlying process has long-range temporal correlations and can fail even for processes generated by finite-memory hidden Markov models. We circumvent the curse of dimensionality in rate-distortion analysis of infinite-order processes by casting predictive rate-distortion objective functions in terms of the forward- and reverse-time causal states of computational mechanics. Examples demonstrate that the resulting causal rate-distortion theory substantially improves current predictive rate-distortion analyses.