In-Context Learning Under Regime Change

TL;DR

This paper formalizes how transformer models can detect and adapt to regime shifts in non-stationary data, providing theoretical constructions and validating them with experiments on synthetic and real-world data.

Contribution

It introduces a formal framework for in-context change-point detection in transformers and demonstrates how model complexity relates to knowledge of change-point timing.

Findings

Transformers can be constructed to solve in-context change-point detection.

Model complexity depends on the level of knowledge about change-point timing.

Pretrained models improve performance on real-world regime change tasks without retraining.

Abstract

Non-stationary sequences arise naturally in control, forecasting, and decision-making. The data-generating process shifts at unknown times, and models must detect the change, discard or downweight obsolete evidence, and adapt to new dynamics on the fly. Transformer-based foundation models increasingly rely on in-context learning for time series forecasting, tabular prediction, and continuous control. As these models are deployed in non-stationary environments, understanding their ability to detect and adapt to regime shifts is important. We formalize this as an in-context change-point detection problem and formally establish the existence of transformer models that solve this problem. Our construction demonstrates that model complexity, in layers and parameters, depends on the level of information available about the change-point location, from no knowledge to knowing exact timing. We validate our results with experiments on synthetic linear regression and linear dynamical systems, where trained transformers match the performance of optimal baselines across information levels. We also show that encoding and incorporating changepoint knowledge indeed improves the real-world performance of a pretrained foundation models on infectious disease forecasting and on financial volatility forecasting around Federal Open Market Committee (FOMC) announcements without retraining, demonstrating practical applicability to real-world regime changes.