Cherry-Picking in Time Series Forecasting: How to Select Datasets to Make Your Model Shine

TL;DR

This study investigates how dataset selection bias, especially cherry-picking, impacts the evaluation of time series forecasting methods, revealing that limited datasets can misrepresent method performance and emphasizing the need for comprehensive testing.

Contribution

It highlights the effects of dataset selection bias on forecasting method evaluation and proposes that increasing dataset diversity improves reliability of performance assessments.

Findings

Cherry-picking datasets can exaggerate method effectiveness.

Using only four datasets, 46% of methods appear top-performing.

Expanding from 3 to 6 datasets reduces incorrect best-method identification by 40%.

Abstract

The importance of time series forecasting drives continuous research and the development of new approaches to tackle this problem. Typically, these methods are introduced through empirical studies that frequently claim superior accuracy for the proposed approaches. Nevertheless, concerns are rising about the reliability and generalizability of these results due to limitations in experimental setups. This paper addresses a critical limitation: the number and representativeness of the datasets used. We investigate the impact of dataset selection bias, particularly the practice of cherry-picking datasets, on the performance evaluation of forecasting methods. Through empirical analysis with a diverse set of benchmark datasets, our findings reveal that cherry-picking datasets can significantly distort the perceived performance of methods, often exaggerating their effectiveness. Furthermore, our results demonstrate that by selectively choosing just four datasets - what most studies report - 46% of methods could be deemed best in class, and 77% could rank within the top three. Additionally, recent deep learning-based approaches show high sensitivity to dataset selection, whereas classical methods exhibit greater robustness. Finally, our results indicate that, when empirically validating forecasting algorithms on a subset of the benchmarks, increasing the number of datasets tested from 3 to 6 reduces the risk of incorrectly identifying an algorithm as the best one by approximately 40%. Our study highlights the critical need for comprehensive evaluation frameworks that more accurately reflect real-world scenarios. Adopting such frameworks will ensure the development of robust and reliable forecasting methods.