A Mechanistic Analysis of Transformers for Dynamical Systems

TL;DR

This paper investigates how single-layer Transformers process dynamical systems, revealing their capabilities and limitations through a dynamical systems perspective, and explaining their success or failure in modeling time-series data.

Contribution

It provides a theoretical analysis of Transformer mechanisms in dynamical systems, connecting empirical observations with classical theory and identifying operational regimes.

Findings

Convexity constraint limits dynamics representation in linear systems.

Attention acts as an adaptive delay-embedding in nonlinear, partially observable systems.

Oversmoothing occurs in oscillatory linear systems due to softmax attention.

Abstract

Transformers are increasingly adopted for modeling and forecasting time-series, yet their internal mechanisms remain poorly understood from a dynamical systems perspective. In contrast to classical autoregressive and state-space models, which benefit from well-established theoretical foundations, Transformer architectures are typically treated as black boxes. This gap becomes particularly relevant as attention-based models are considered for general-purpose or zero-shot forecasting across diverse dynamical regimes. In this work, we do not propose a new forecasting model, but instead investigate the representational capabilities and limitations of single-layer Transformers when applied to dynamical data. Building on a dynamical systems perspective we interpret causal self-attention as a linear, history-dependent recurrence and analyze how it processes temporal information. Through a series of linear and nonlinear case studies, we identify distinct operational regimes. For linear systems, we show that the convexity constraint imposed by softmax attention fundamentally restricts the class of dynamics that can be represented, leading to oversmoothing in oscillatory settings. For nonlinear systems under partial observability, attention instead acts as an adaptive delay-embedding mechanism, enabling effective state reconstruction when sufficient temporal context and latent dimensionality are available. These results help bridge empirical observations with classical dynamical systems theory, providing insight into when and why Transformers succeed or fail as models of dynamical systems.