Forecasting with Deep Learning: Beyond Average of Average of Average Performance

TL;DR

This paper introduces a multi-perspective evaluation framework for forecasting models, revealing that deep learning methods like NHITS outperform classical techniques mainly in multi-step ahead forecasting, with performance varying across conditions.

Contribution

It proposes a novel evaluation framework that considers different forecasting aspects, demonstrating the importance of nuanced assessment over single-score metrics.

Findings

NHITS generally outperforms classical methods

Performance advantage of NHITS depends on forecasting horizon

NHITS is less effective when dealing with anomalies

Abstract

Accurate evaluation of forecasting models is essential for ensuring reliable predictions. Current practices for evaluating and comparing forecasting models focus on summarising performance into a single score, using metrics such as SMAPE. We hypothesize that averaging performance over all samples dilutes relevant information about the relative performance of models. Particularly, conditions in which this relative performance is different than the overall accuracy. We address this limitation by proposing a novel framework for evaluating univariate time series forecasting models from multiple perspectives, such as one-step ahead forecasting versus multi-step ahead forecasting. We show the advantages of this framework by comparing a state-of-the-art deep learning approach with classical forecasting techniques. While classical methods (e.g. ARIMA) are long-standing approaches to forecasting, deep neural networks (e.g. NHITS) have recently shown state-of-the-art forecasting performance in benchmark datasets. We conducted extensive experiments that show NHITS generally performs best, but its superiority varies with forecasting conditions. For instance, concerning the forecasting horizon, NHITS only outperforms classical approaches for multi-step ahead forecasting. Another relevant insight is that, when dealing with anomalies, NHITS is outperformed by methods such as Theta. These findings highlight the importance of aspect-based model evaluation.