Compressed Predictive Information Coding

TL;DR

This paper introduces CPIC, an information-theoretic framework that learns low-dimensional, predictive representations from dynamic data by minimizing compression complexity and maximizing predictive information, with applications in neuroscience and noisy systems.

Contribution

The paper proposes a novel variational framework, CPIC, for extracting predictive representations from dynamic data, outperforming existing methods especially in noisy environments.

Findings

CPIC effectively recovers latent spaces in noisy dynamical systems.

Stochastic encoders improve the robustness of learned representations.

Variational bounds of mutual information enhance the tractability of the method.

Abstract

Unsupervised learning plays an important role in many fields, such as artificial intelligence, machine learning, and neuroscience. Compared to static data, methods for extracting low-dimensional structure for dynamic data are lagging. We developed a novel information-theoretic framework, Compressed Predictive Information Coding (CPIC), to extract useful representations from dynamic data. CPIC selectively projects the past (input) into a linear subspace that is predictive about the compressed data projected from the future (output). The key insight of our framework is to learn representations by minimizing the compression complexity and maximizing the predictive information in latent space. We derive variational bounds of the CPIC loss which induces the latent space to capture information that is maximally predictive. Our variational bounds are tractable by leveraging bounds of mutual information. We find that introducing stochasticity in the encoder robustly contributes to better representation. Furthermore, variational approaches perform better in mutual information estimation compared with estimates under a Gaussian assumption. We demonstrate that CPIC is able to recover the latent space of noisy dynamical systems with low signal-to-noise ratios, and extracts features predictive of exogenous variables in neuroscience data.