Deep Learning-based Approaches for State Space Models: A Selective Review

TL;DR

This paper reviews recent deep learning methods for state-space models, highlighting their applications in sequence modeling and irregular time series, and discusses classical and modern approaches including neural differential equations.

Contribution

It provides a unified perspective on deep neural network-based state-space models, covering both discrete and continuous time frameworks, and discusses recent innovations and applications.

Findings

Deep neural approaches enhance SSM flexibility and efficiency.

SSMs effectively handle irregular and mixed frequency time series.

Recent models improve sequence modeling performance.

Abstract

State-space models (SSMs) offer a powerful framework for dynamical system analysis, wherein the temporal dynamics of the system are assumed to be captured through the evolution of the latent states, which govern the values of the observations. This paper provides a selective review of recent advancements in deep neural network-based approaches for SSMs, and presents a unified perspective for discrete time deep state space models and continuous time ones such as latent neural Ordinary Differential and Stochastic Differential Equations. It starts with an overview of the classical maximum likelihood based approach for learning SSMs, reviews variational autoencoder as a general learning pipeline for neural network-based approaches in the presence of latent variables, and discusses in detail representative deep learning models that fall under the SSM framework. Very recent developments, where SSMs are used as standalone architectural modules for improving efficiency in sequence modeling, are also examined. Finally, examples involving mixed frequency and irregularly-spaced time series data are presented to demonstrate the advantage of SSMs in these settings.