On the Internal Semantics of Time-Series Foundation Models

TL;DR

This paper systematically investigates how Time-series Foundation Models internally represent concepts, revealing that different layers encode various temporal features and that compositional understanding remains challenging.

Contribution

It provides a detailed analysis of concept encoding, linear recoverability, and representation evolution in TSFMs, highlighting current limitations in compositional understanding.

Findings

Early layers encode local time-domain patterns

Deeper layers capture dispersion and change signals

Representation of concept composition degrades with complexity

Abstract

Time-series Foundation Models (TSFMs) have recently emerged as a universal paradigm for learning across diverse temporal domains. However, despite their empirical success, the internal mechanisms by which these models represent fundamental time-series concepts remain poorly understood. In this work, we undertake a systematic investigation of concept interpretability in TSFMs. Specifically, we examine: (i) which layers encode which concepts, (ii) whether concept parameters are linearly recoverable, (iii) how representations evolve in terms of concept disentanglement and abstraction across model depth, and (iv) how models process compositions of concepts. We systematically probe these questions using layer-wise analyses, linear recoverability tests, and representation similarity measures, providing a structured account of TSFM semantics. The resulting insights show that early layers mainly capture local, time-domain patterns (e.g., AR(1), level shifts, trends), while deeper layers encode dispersion and change-time signals, with spectral and warping factors remaining the hardest to recover linearly. In compositional settings, however, probe performance degrades, revealing interference between concepts. This highlights that while atomic concepts are reliably localized, composition remains a challenge, underscoring a key limitation in current TSFMs' ability to represent interacting temporal phenomena.