SynTSBench: Rethinking Temporal Pattern Learning in Deep Learning Models for Time Series

TL;DR

SynTSBench introduces a synthetic data-driven evaluation framework for time series forecasting models, enabling systematic, interpretable assessment of their pattern learning, robustness, and theoretical limits to improve model selection and development.

Contribution

This paper presents SynTSBench, a novel evaluation paradigm that isolates confounding factors and provides a comprehensive, interpretable assessment of deep learning models' capabilities in time series forecasting.

Findings

Current models do not consistently approach optimal performance across all pattern types.

SynTSBench effectively isolates model strengths and weaknesses in pattern learning.

The framework highlights the gap between model predictions and theoretical optima.

Abstract

Recent advances in deep learning have driven rapid progress in time series forecasting, yet many state-of-the-art models continue to struggle with robust performance in real-world applications, even when they achieve strong results on standard benchmark datasets. This persistent gap can be attributed to the black-box nature of deep learning architectures and the inherent limitations of current evaluation frameworks, which frequently lack the capacity to provide clear, quantitative insights into the specific strengths and weaknesses of different models, thereby complicating the selection of appropriate models for particular forecasting scenarios. To address these issues, we propose a synthetic data-driven evaluation paradigm, SynTSBench, that systematically assesses fundamental modeling capabilities of time series forecasting models through programmable feature configuration. Our framework isolates confounding factors and establishes an interpretable evaluation system with three core analytical dimensions: (1) temporal feature decomposition and capability mapping, which enables systematic evaluation of model capacities to learn specific pattern types; (2) robustness analysis under data irregularities, which quantifies noise tolerance thresholds and anomaly recovery capabilities; and (3) theoretical optimum benchmarking, which establishes performance boundaries for each pattern type-enabling direct comparison between model predictions and mathematical optima. Our experiments show that current deep learning models do not universally approach optimal baselines across all types of temporal features.The code is available at https://github.com/TanQitai/SynTSBench