TSFM-Bench: A Comprehensive and Unified Benchmark of Foundation Models for Time Series Forecasting

TL;DR

This paper introduces TSFM-Bench, a comprehensive benchmark for evaluating Time Series Foundation Models across diverse datasets, scenarios, and evaluation protocols, aiming to advance generalizable forecasting methods.

Contribution

It presents a unified benchmarking framework for TSFMs, supporting multiple scenarios and standardized evaluation protocols, to facilitate fair comparison and drive future model development.

Findings

Extensive evaluation of existing TSFMs across diverse datasets.

Identification of strengths and limitations of current TSFMs.

Proposals for future directions in model design.

Abstract

Time Series Forecasting (TSF) is key functionality in numerous fields, such as financial investment, weather services, and energy management. Although increasingly capable TSF methods occur, many of them require domain-specific data collection and model training and do not generalize well when applied in other domains. Time Series Foundation Models (TSFMs) that are pre-trained on massive heterogeneous time series data aim to overcome these limitations. The prospects for generalizability have spurred the development of a new generation of TSFMs. This study proposes a benchmark, TSFM-Bench, to facilitate comprehensive and unified evaluation of TSFMs. TSFM-Bench covers a wide range of TSFMs, including those based on large language models and those pre-trained on time series data. TSFM-Bench supports multiple forecasting scenarios, including zero-shot, few-shot, and full-shot, enabling assessment across the full range of adaptation strategies. TSFM-Bench also provides a standardized experimental protocols for critical evaluation processes such as dataset splitting, loading, normalization, and few-shot sampling, facilitating consistency and fairness. We report on an extensive evaluation of TSFMs across a diverse range of datasets spanning multiple domains and exhibiting varied statistical characteristics. Specifically, we identify pros and cons and inherent limitations of existing TSFMs, and we propose potential directions for new model designs.