Integrating Multimodal Data for Joint Generative Modeling of Complex Dynamics

TL;DR

This paper introduces a flexible, generative deep learning framework that integrates multimodal data for reconstructing complex nonlinear dynamical systems, enabling reconstruction from diverse data types including symbolic labels.

Contribution

It presents a novel multimodal variational autoencoder-based approach for dynamical systems reconstruction that unifies different data modalities within a generative modeling framework.

Findings

Enables reconstruction from symbolic data alone

Connects various observation types in a shared latent space

Produces trajectories with true geometrical and temporal structure

Abstract

Many, if not most, systems of interest in science are naturally described as nonlinear dynamical systems. Empirically, we commonly access these systems through time series measurements. Often such time series may consist of discrete random variables rather than continuous measurements, or may be composed of measurements from multiple data modalities observed simultaneously. For instance, in neuroscience we may have behavioral labels in addition to spike counts and continuous physiological recordings. While by now there is a burgeoning literature on deep learning for dynamical systems reconstruction (DSR), multimodal data integration has hardly been considered in this context. Here we provide such an efficient and flexible algorithmic framework that rests on a multimodal variational autoencoder for generating a sparse teacher signal that guides training of a reconstruction model, exploiting recent advances in DSR training techniques. It enables to combine various sources of information for optimal reconstruction, even allows for reconstruction from symbolic data (class labels) alone, and connects different types of observations within a common latent dynamics space. In contrast to previous multimodal data integration techniques for scientific applications, our framework is fully \textit{generative}, producing, after training, trajectories with the same geometrical and temporal structure as those of the ground truth system.