Time-Delayed Transformers for Data-Driven Modeling of Low-Dimensional Dynamics

TL;DR

The paper introduces a minimalistic transformer architecture called TD-TF for modeling complex unsteady spatio-temporal dynamics, bridging linear methods and deep sequence models with efficient computation.

Contribution

It presents a simplified transformer model that generalizes TD-DMD, offering improved modeling of nonlinear and chaotic systems with interpretability and efficiency.

Findings

Matches linear baselines on near-linear systems

Outperforms linear models on nonlinear and chaotic regimes

Accurately captures long-term dynamics in complex systems

Abstract

We propose the time-delayed transformer (TD-TF), a simplified transformer architecture for data-driven modeling of unsteady spatio-temporal dynamics. TD-TF bridges linear operator-based methods and deep sequence models by showing that a single-layer, single-head transformer can be interpreted as a nonlinear generalization of time-delayed dynamic mode decomposition (TD-DMD). The architecture is deliberately minimal, consisting of one self-attention layer with a single query per prediction and one feedforward layer, resulting in linear computational complexity in sequence length and a small parameter count. Numerical experiments demonstrate that TD-TF matches the performance of strong linear baselines on near-linear systems, while significantly outperforming them in nonlinear and chaotic regimes, where it accurately captures long-term dynamics. Validation studies on synthetic signals, unsteady aerodynamics, the Lorenz '63 system, and a reaction-diffusion model show that TD-TF preserves the interpretability and efficiency of linear models while providing substantially enhanced expressive power for complex dynamics.