Non-Stationarity in the Embedding Space of Time Series Foundation Models

TL;DR

This paper investigates how various forms of non-stationarity in time series are represented and detectable in the embedding spaces of foundation models, revealing model-specific detection capabilities and failure modes.

Contribution

It provides a systematic analysis of non-stationarity detection in TSFM embeddings, distinguishing distributional shifts from persistence-related non-stationarity, and highlights model-specific behaviors.

Findings

Embedding-space detectability of non-stationarity degrades smoothly with shift strength.

Different TSFMs exhibit distinct failure modes in detecting non-stationarity.

Linear forms of non-stationarity are accessible under controlled conditions.

Abstract

Time series foundation models (TSFMs) are widely used as generic feature extractors, yet the notion of non-stationarity in their embedding spaces remains poorly understood. Recent work often conflates non-stationarity with distribution shift, blurring distinctions fundamental to classical time-series analysis and long-standing methodologies such as statistical process control (SPC). In SPC, non-stationarity signals a process leaving a stable regime - via shifts in mean, variance, or emerging trends - and detecting such departures is central to quality monitoring and change-point analysis. Motivated by this diagnostic tradition, we study how different forms of distributional non-stationarity - mean shifts, variance changes, and linear trends - become linearly accessible in TSFM embedding spaces under controlled conditions. We further examine temporal non-stationarity arising from persistence, which reflects violations of weak stationarity due to long-memory or near-unit-root behavior rather than explicit distributional shifts. By sweeping shift strength and probing multiple TSFMs, we find that embedding-space detectability of non-stationarity degrades smoothly and that different models exhibit distinct, model-specific failure modes.