It's TIME: Towards the Next Generation of Time Series Forecasting Benchmarks

TL;DR

This paper introduces TIME, a comprehensive, high-quality benchmark with diverse datasets and tasks for evaluating time series foundation models, emphasizing real-world relevance and generalizable insights.

Contribution

It presents a new benchmark with 50 datasets and 98 tasks, rigorous data quality assurance, real-world task formulation, and a pattern-level evaluation approach for better model assessment.

Findings

Evaluated 12 TSFMs on the new benchmark.

Established a multi-granular leaderboard for analysis.

Demonstrated improved insights through pattern-level evaluation.

Abstract

Time series foundation models (TSFMs) are revolutionizing the forecasting landscape from specific dataset modeling to generalizable task evaluation. However, we contend that existing benchmarks exhibit common limitations in four dimensions: constrained data composition dominated by reused legacy sources, compromised data integrity lacking rigorous quality assurance, misaligned task formulations detached from real-world contexts, and rigid analysis perspectives that obscure generalizable insights. To bridge these gaps, we introduce TIME, a next-generation task-centric benchmark comprising 50 fresh datasets and 98 forecasting tasks, tailored for strict zero-shot TSFM evaluation free from data leakage. Integrating large language models and human expertise, we establish a rigorous human-in-the-loop benchmark construction pipeline to ensure high data integrity and redefine task formulation by aligning forecasting configurations with real-world operational requirements and variate predictability. Furthermore, we propose a novel pattern-level evaluation perspective that moves beyond traditional dataset-level evaluations based on static meta labels. By leveraging structural time series features to characterize intrinsic temporal properties, this approach offers generalizable insights into model capabilities across diverse patterns. We evaluate 12 representative TSFMs and establish a multi-granular leaderboard to facilitate in-depth analysis and visualized inspection. The leaderboard is available at https://huggingface.co/spaces/Real-TSF/TIME-leaderboard.