InfoDPCCA: Information-Theoretic Dynamic Probabilistic Canonical Correlation Analysis

TL;DR

InfoDPCCA is a novel information-theoretic dynamic CCA framework that extracts shared and sequence-specific latent representations from high-dimensional sequential data, improving interpretability and robustness in applications like medical imaging.

Contribution

We propose InfoDPCCA, a new dynamic probabilistic CCA model that explicitly encodes mutual information in the shared latent space, with a novel training scheme and stability mechanisms.

Findings

Outperforms prior models on synthetic data

Effectively captures shared information in fMRI data

Enhances interpretability and robustness

Abstract

Extracting meaningful latent representations from high-dimensional sequential data is a crucial challenge in machine learning, with applications spanning natural science and engineering. We introduce InfoDPCCA, a dynamic probabilistic Canonical Correlation Analysis (CCA) framework designed to model two interdependent sequences of observations. InfoDPCCA leverages a novel information-theoretic objective to extract a shared latent representation that captures the mutual structure between the data streams and balances representation compression and predictive sufficiency while also learning separate latent components that encode information specific to each sequence. Unlike prior dynamic CCA models, such as DPCCA, our approach explicitly enforces the shared latent space to encode only the mutual information between the sequences, improving interpretability and robustness. We further introduce a two-step training scheme to bridge the gap between information-theoretic representation learning and generative modeling, along with a residual connection mechanism to enhance training stability. Through experiments on synthetic and medical fMRI data, we demonstrate that InfoDPCCA excels as a tool for representation learning. Code of InfoDPCCA is available at https://github.com/marcusstang/InfoDPCCA.