Transformers as Implicit State Estimators: In-Context Learning in Dynamical Systems

TL;DR

This paper demonstrates that transformers can implicitly perform state estimation in dynamical systems through in-context learning, matching traditional filters in linear cases and approaching their performance in nonlinear regimes without explicit model knowledge.

Contribution

It introduces a novel perspective that transformers can serve as implicit, non-parametric state estimators for dynamical systems via in-context learning, bypassing traditional filtering methods.

Findings

Transformers accurately predict outputs using in-context learning without explicit model updates.

Performance closely matches Kalman filter in linear-Gaussian systems.

Approaches EKF and PF performance in nonlinear regimes.

Abstract

Predicting the behavior of a dynamical system from noisy observations of its past outputs is a classical problem encountered across engineering and science. For linear systems with Gaussian inputs, the Kalman filter -- the best linear minimum mean-square error estimator of the state trajectory -- is optimal in the Bayesian sense. For nonlinear systems, Bayesian filtering is typically approached using suboptimal heuristics such as the Extended Kalman Filter (EKF), or numerical methods such as particle filtering (PF). In this work, we show that transformers, employed in an in-context learning (ICL) setting, can implicitly infer hidden states in order to predict the outputs of a wide family of dynamical systems, without test-time gradient updates or explicit knowledge of the system model. Specifically, when provided with a short context of past input-output pairs and, optionally, system parameters, a frozen transformer accurately predicts the current output. In linear-Gaussian regimes, its predictions closely match those of the Kalman filter; in nonlinear regimes, its performance approaches that of EKF and PF. Moreover, prediction accuracy degrades gracefully when key parameters, such as the state-transition matrix, are withheld from the context, demonstrating robustness and implicit parameter inference. These findings suggest that transformer in-context learning provides a flexible, non-parametric alternative for output prediction in dynamical systems, grounded in implicit latent-state estimation.