LyTimeT: Towards Robust and Interpretable State-Variable Discovery

TL;DR

LyTimeT introduces a two-phase framework that extracts robust, interpretable latent variables from high-dimensional videos of dynamical systems, effectively handling distractions and ensuring physical interpretability.

Contribution

The paper presents a novel two-phase approach combining spatio-temporal attention and stability regularization for interpretable state-variable discovery from videos.

Findings

Achieves accurate long-horizon predictions on synthetic and real-world systems.

Estimates mutual information and intrinsic dimension close to ground truth.

Remains invariant under background perturbations and outperforms baselines.

Abstract

Extracting the true dynamical variables of a system from high-dimensional video is challenging due to distracting visual factors such as background motion, occlusions, and texture changes. We propose LyTimeT, a two-phase framework for interpretable variable extraction that learns robust and stable latent representations of dynamical systems. In Phase 1, LyTimeT employs a spatio-temporal TimeSformer-based autoencoder that uses global attention to focus on dynamically relevant regions while suppressing nuisance variation, enabling distraction-robust latent state learning and accurate long-horizon video prediction. In Phase 2, we probe the learned latent space, select the most physically meaningful dimensions using linear correlation analysis, and refine the transition dynamics with a Lyapunov-based stability regularizer to enforce contraction and reduce error accumulation during roll-outs. Experiments on five synthetic benchmarks and four real-world dynamical systems, including chaotic phenomena, show that LyTimeT achieves mutual information and intrinsic dimension estimates closest to ground truth, remains invariant under background perturbations, and delivers the lowest analytical mean squared error among CNN-based (TIDE) and transformer-only baselines. Our results demonstrate that combining spatio-temporal attention with stability constraints yields predictive models that are not only accurate but also physically interpretable.