Dynamical modeling of nonlinear latent factors in multiscale neural activity with real-time inference

TL;DR

This paper introduces a novel real-time decoding framework for multimodal neural data that effectively handles different timescales, distributions, and missing samples, improving decoding accuracy in neuroscience applications.

Contribution

The authors develop a multiscale, nonlinear modeling framework enabling real-time recursive decoding across diverse neural modalities with different sampling rates and missing data.

Findings

Outperforms existing benchmarks in real-time decoding accuracy.

Effectively handles missing samples and diverse distributions across modalities.

Validated on multiple neural datasets with improved results.

Abstract

Real-time decoding of target variables from multiple simultaneously recorded neural time-series modalities, such as discrete spiking activity and continuous field potentials, is important across various neuroscience applications. However, a major challenge for doing so is that different neural modalities can have different timescales (i.e., sampling rates) and different probabilistic distributions, or can even be missing at some time-steps. Existing nonlinear models of multimodal neural activity do not address different timescales or missing samples across modalities. Further, some of these models do not allow for real-time decoding. Here, we develop a learning framework that can enable real-time recursive decoding while nonlinearly aggregating information across multiple modalities with different timescales and distributions and with missing samples. This framework consists of 1) a multiscale encoder that nonlinearly aggregates information after learning within-modality dynamics to handle different timescales and missing samples in real time, 2) a multiscale dynamical backbone that extracts multimodal temporal dynamics and enables real-time recursive decoding, and 3) modality-specific decoders to account for different probabilistic distributions across modalities. In both simulations and three distinct multiscale brain datasets, we show that our model can aggregate information across modalities with different timescales and distributions and missing samples to improve real-time target decoding. Further, our method outperforms various linear and nonlinear multimodal benchmarks in doing so.