Understanding the Implicit Biases of Design Choices for Time Series Foundation Models

TL;DR

This paper investigates how various design choices in time series foundation models influence their implicit biases and behavior, combining theory and experiments to understand their effects on model properties and data interactions.

Contribution

It provides a systematic analysis of how training design choices induce implicit biases in TSFMs, offering insights into their effects on model behavior and data interaction.

Findings

Design choices affect temporal and geometric properties of models.

Implicit biases can be intuitive or counterintuitive.

Multiple biases interact complexly in outlier handling.

Abstract

Time series foundation models (TSFMs) are a class of potentially powerful, general-purpose tools for time series forecasting and related temporal tasks, but their behavior is strongly shaped by subtle inductive biases in their design. Rather than developing a new model and claiming that it is better than existing TSFMs, e.g., by winning on existing well-established benchmarks, our objective is to understand how the various ``knobs'' of the training process affect model quality. Using a mix of theory and controlled empirical evaluation, we identify several design choices (patch size, embedding choice, training objective, etc.) and show how they lead to implicit biases in fundamental model properties (temporal behavior, geometric structure, how aggressively or not the model regresses to the mean, etc.); and we show how these biases can be intuitive or very counterintuitive, depending on properties of the model and data. We also illustrate in a case study on outlier handling how multiple biases can interact in complex ways; and we discuss implications of our results for learning the bitter lesson and building TSFMs.