Continuous Spatiotemporal Transformers

TL;DR

The paper introduces the Continuous Spatiotemporal Transformer (CST), a novel architecture designed to model continuous dynamical systems with guarantees of smooth outputs, outperforming traditional methods in various synthetic and real-world tasks.

Contribution

It proposes a new transformer architecture that models continuous systems and ensures smooth outputs through Sobolev space optimization, addressing limitations of discrete models.

Findings

CST outperforms traditional transformers in synthetic and real system benchmarks.

CST effectively learns brain dynamics from calcium imaging data.

The framework guarantees continuous and smooth outputs for dynamical systems.

Abstract

Modeling spatiotemporal dynamical systems is a fundamental challenge in machine learning. Transformer models have been very successful in NLP and computer vision where they provide interpretable representations of data. However, a limitation of transformers in modeling continuous dynamical systems is that they are fundamentally discrete time and space models and thus have no guarantees regarding continuous sampling. To address this challenge, we present the Continuous Spatiotemporal Transformer (CST), a new transformer architecture that is designed for the modeling of continuous systems. This new framework guarantees a continuous and smooth output via optimization in Sobolev space. We benchmark CST against traditional transformers as well as other spatiotemporal dynamics modeling methods and achieve superior performance in a number of tasks on synthetic and real systems, including learning brain dynamics from calcium imaging data.