RHiOTS: A Framework for Evaluating Hierarchical Time Series Forecasting Algorithms

TL;DR

RHiOTS is a comprehensive framework that evaluates the robustness of hierarchical time series forecasting algorithms under various data transformations, revealing insights into their reliability and behavior in real-world scenarios.

Contribution

The paper introduces RHiOTS, a novel framework that systematically assesses the robustness of forecasting models through dataset modifications and visualization tools.

Findings

Statistical methods are generally more robust than deep learning algorithms.

Robustness varies significantly with data transformations.

Reconciliation methods like MinT do not significantly affect robustness.

Abstract

We introduce the Robustness of Hierarchically Organized Time Series (RHiOTS) framework, designed to assess the robustness of hierarchical time series forecasting models and algorithms on real-world datasets. Hierarchical time series, where lower-level forecasts must sum to upper-level ones, are prevalent in various contexts, such as retail sales across countries. Current empirical evaluations of forecasting methods are often limited to a small set of benchmark datasets, offering a narrow view of algorithm behavior. RHiOTS addresses this gap by systematically altering existing datasets and modifying the characteristics of individual series and their interrelations. It uses a set of parameterizable transformations to simulate those changes in the data distribution. Additionally, RHiOTS incorporates an innovative visualization component, turning complex, multidimensional robustness evaluation results into intuitive, easily interpretable visuals. This approach allows an in-depth analysis of algorithm and model behavior under diverse conditions. We illustrate the use of RHiOTS by analyzing the predictive performance of several algorithms. Our findings show that traditional statistical methods are more robust than state-of-the-art deep learning algorithms, except when the transformation effect is highly disruptive. Furthermore, we found no significant differences in the robustness of the algorithms when applying specific reconciliation methods, such as MinT. RHiOTS provides researchers with a comprehensive tool for understanding the nuanced behavior of forecasting algorithms, offering a more reliable basis for selecting the most appropriate method for a given problem.