Can Transformers Learn Optimal Filtering for Unknown Systems?

TL;DR

This paper explores the use of transformer models for optimal filtering in dynamical systems, demonstrating their ability to adapt to unseen systems and match traditional filters like Kalman filters, with statistical guarantees and identified limitations.

Contribution

It introduces a novel application of transformers for system estimation, showing their effectiveness across linear and nonlinear dynamics with theoretical guarantees.

Findings

Transformers match Kalman filter performance on linear systems.

Transformers perform well on complex nonlinear, time-varying systems.

Statistical bounds on training data needed for desired accuracy.

Abstract

Transformer models have shown great success in natural language processing; however, their potential remains mostly unexplored for dynamical systems. In this work, we investigate the optimal output estimation problem using transformers, which generate output predictions using all the past ones. Particularly, we train the transformer using various distinct systems and then evaluate the performance on unseen systems with unknown dynamics. Empirically, the trained transformer adapts exceedingly well to different unseen systems and even matches the optimal performance given by the Kalman filter for linear systems. In more complex settings with non-i.i.d. noise, time-varying dynamics, and nonlinear dynamics like a quadrotor system with unknown parameters, transformers also demonstrate promising results. To support our experimental findings, we provide statistical guarantees that quantify the amount of training data required for the transformer to achieve a desired excess risk. Finally, we point out some limitations by identifying two classes of problems that lead to degraded performance, highlighting the need for caution when using transformers for control and estimation.