Channel Dependence, Limited Lookback Windows, and the Simplicity of Datasets: How Biased is Time Series Forecasting?

TL;DR

This paper investigates how dataset simplicity and channel dependencies affect time series forecasting, emphasizing the importance of tuning lookback windows and choosing models based on dataset characteristics.

Contribution

It highlights the impact of dataset bias and channel dependence on model performance, advocating for tailored hyperparameter tuning and dataset analysis in TSF.

Findings

Channel-independent models perform well on simple datasets due to weak inter-channel correlations.

Multivariate channel-dependent models excel on datasets with strong cross-channel dependencies.

Proper tuning of lookback windows is crucial for fair and accurate model evaluation.

Abstract

In Long-term Time Series Forecasting (LTSF), the lookback window is a critical hyperparameter often set arbitrarily, undermining the validity of model evaluations. We argue that the lookback window must be tuned on a per-task basis to ensure fair comparisons. Our empirical results show that failing to do so can invert performance rankings, particularly when comparing univariate and multivariate methods. Experiments on standard benchmarks reposition Channel-Independent (CI) models, such as PatchTST, as state-of-the-art methods. However, we reveal this superior performance is largely an artifact of weak inter-channel correlations and simplicity of patterns within these specific datasets. Using Granger causality analysis and ODE datasets (with implicit channel correlations), we demonstrate that the true strength of multivariate Channel-Dependent (CD) models emerges on datasets with strong, inherent cross-channel dependencies, where they significantly outperform CI models. We conclude with four key recommendations for improving TSF research: (i) consider the lookback window as a key hyperparameter to tune, (ii) for standard datasets, examining CI architectures is advantageous, (iii) leverage statistical analysis of datasets to guide the choice between CI and CD architectures, and (iv) prefer CD models in scenarios with limited data.