Online detection of forecast model inadequacies using forecast errors

TL;DR

This paper introduces a real-time monitoring framework using sequential changepoint detection on forecast errors to identify process changes affecting forecast accuracy, demonstrated through simulations and real-world examples.

Contribution

It presents a novel online monitoring method that detects forecast model inadequacies faster than traditional approaches by analyzing forecast errors for process changes.

Findings

Effective in detecting process changes in simulations

Outperforms alternative methods in real-world case studies

Applicable across diverse forecasting models

Abstract

In many organisations, accurate forecasts are essential for making informed decisions for a variety of applications from inventory management to staffing optimization. Whatever forecasting model is used, changes in the underlying process can lead to inaccurate forecasts, which will be damaging to decision-making. At the same time, models are becoming increasingly complex and identifying change through direct modelling is problematic. We present a novel framework for online monitoring of forecasts to ensure they remain accurate. By utilizing sequential changepoint techniques on the forecast errors, our framework allows for the real-time identification of potential changes in the process caused by various external factors. We show theoretically that some common changes in the underlying process will manifest in the forecast errors and can be identified faster by identifying shifts in the forecast errors than within the original modelling framework. Moreover, we demonstrate the effectiveness of this framework on numerous forecasting approaches through simulations and show its effectiveness over alternative approaches. Finally, we present two concrete examples, one from Royal Mail parcel delivery volumes and one from NHS A\&E admissions relating to gallstones.