QuitoBench: A High-Quality Open Time Series Forecasting Benchmark

TL;DR

QuitoBench introduces a comprehensive, high-quality benchmark for time series forecasting, enabling better evaluation of models across diverse regimes and revealing insights into model performance and scaling effects.

Contribution

The paper presents QuitoBench, a large-scale, regime-balanced benchmark built on a billion-scale dataset, facilitating more nuanced and reproducible evaluation of forecasting models.

Findings

Deep learning models outperform foundation models at short contexts; foundation models excel at long contexts.

Forecastability significantly impacts forecasting difficulty, with a 3.64× MAE gap.

Deep learning models achieve comparable or better performance with 59× fewer parameters.

Abstract

Time series forecasting is critical across finance, healthcare, and cloud computing, yet progress is constrained by a fundamental bottleneck: the scarcity of large-scale, high-quality benchmarks. To address this gap, we introduce \textsc{QuitoBench}, a regime-balanced benchmark for time series forecasting with coverage across eight trend$\times$seasonality$\times$forecastability (TSF) regimes, designed to capture forecasting-relevant properties rather than application-defined domain labels. The benchmark is built upon \textsc{Quito}, a billion-scale time series corpus of application traffic from Alipay spanning nine business domains. Benchmarking 10 models from deep learning, foundation models, and statistical baselines across 232,200 evaluation instances, we report four key findings: (i) a context-length crossover where deep learning models lead at short context ($L=96$) but foundation models dominate at long context ($L \ge 576$); (ii) forecastability is the dominant difficulty driver, producing a $3.64 \times$ MAE gap across regimes; (iii) deep learning models match or surpass foundation models at $59 \times$ fewer parameters; and (iv) scaling the amount of training data provides substantially greater benefit than scaling model size for both model families. These findings are validated by strong cross-benchmark and cross-metric consistency. Our open-source release enables reproducible, regime-aware evaluation for time series forecasting research.