Are We Winning the Wrong Game? Revisiting Evaluation Practices for Long-Term Time Series Forecasting

TL;DR

This paper critiques current long-term time series forecasting evaluation practices, arguing they focus too narrowly on error metrics and proposing a multi-dimensional approach that better captures real-world forecasting objectives.

Contribution

It introduces a multi-dimensional evaluation framework for LTSF that emphasizes structural fidelity and decision relevance over traditional pointwise error metrics.

Findings

Current benchmarks may mislead progress assessment

Proposed evaluation considers structural and decision-oriented metrics

Highlights misalignment between metrics and real-world forecasting goals

Abstract

Long-term time series forecasting (LTSF) is widely recognized as a central challenge in data mining and machine learning. LTSF has increasingly evolved into a benchmark-driven ''GAME,'' where models are ranked, compared, and declared state-of-the-art based primarily on marginal reductions in aggregated pointwise error metrics such as MSE and MAE. Across a small set of canonical datasets and fixed forecasting horizons, progress is communicated through leaderboard-style tables in which lower numerical scores define success. In this GAME, what is measured becomes what is optimized, and incremental error reduction becomes the dominant currency of advancement. We argue that this metric-centric regime is not merely incomplete, but structurally misaligned with the broader objectives of forecasting. In real-world settings, forecasting often prioritizes preserving temporal structure, trend stability, seasonal coherence, robustness to regime shifts, and supporting downstream decision processes. Optimizing aggregate pointwise error does not necessarily imply modeling these structural properties. As a result, leaderboard improvement may increasingly reflect specialization in benchmark configurations rather than a deeper understanding of temporal dynamics. This paper revisits LTSF evaluation as a foundational question in data science: what does it mean to measure forecasting progress? We propose a multi-dimensional evaluation perspective that integrates statistical fidelity, structural coherence, and decision-level relevance. By challenging the current metric monoculture, we aim to redirect attention from winning benchmark tables toward advancing meaningful, context-aware forecasting.