Learning Latent Dynamics via Invariant Decomposition and (Spatio-)Temporal Transformers

TL;DR

This paper introduces a scalable method combining variational autoencoders and (spatio-)temporal attention to learn invariant latent dynamics from heterogeneous high-dimensional data without explicit neural ODEs.

Contribution

It presents a novel, data-driven approach that separates instance-specific encodings from universal latent dynamics, enabling efficient, scalable learning from multiple system instances.

Findings

Outperforms state-of-the-art neural dynamical models on synthetic and real data.

Effectively infers continuous-time system behavior without neural ODEs.

Demonstrates transfer learning to new system interventions.

Abstract

We propose a method for learning dynamical systems from high-dimensional empirical data that combines variational autoencoders and (spatio-)temporal attention within a framework designed to enforce certain scientifically-motivated invariances. We focus on the setting in which data are available from multiple different instances of a system whose underlying dynamical model is entirely unknown at the outset. The approach rests on a separation into an instance-specific encoding (capturing initial conditions, constants etc.) and a latent dynamics model that is itself universal across all instances/realizations of the system. The separation is achieved in an automated, data-driven manner and only empirical data are required as inputs to the model. The approach allows effective inference of system behaviour at any continuous time but does not require an explicit neural ODE formulation, which makes it efficient and highly scalable. We study behaviour through simple theoretical analyses and extensive experiments on synthetic and real-world datasets. The latter investigate learning the dynamics of complex systems based on finite data and show that the proposed approach can outperform state-of-the-art neural-dynamical models. We study also more general inductive bias in the context of transfer to data obtained under entirely novel system interventions. Overall, our results provide a promising new framework for efficiently learning dynamical models from heterogeneous data with potential applications in a wide range of fields including physics, medicine, biology and engineering.