Analyzing the retraining frequency of global forecasting models: towards more stable forecasting systems

TL;DR

This paper investigates how the frequency of retraining global forecasting models affects forecast stability, proposing a new stability metric and demonstrating that less frequent retraining can enhance stability without sacrificing accuracy.

Contribution

It introduces a novel, model-agnostic metric for probabilistic forecast stability and provides empirical evidence that infrequent retraining can improve stability and maintain accuracy.

Findings

Less frequent retraining often improves forecast stability.

Stability and accuracy are not necessarily conflicting objectives.

A new metric (SMQC) effectively measures probabilistic forecast stability.

Abstract

Forecast stability, that is, the consistency of predictions over time, is essential in business settings where sudden shifts in forecasts can disrupt planning and erode trust in predictive systems. Despite its importance, stability is often overlooked in favor of accuracy. In this study, we evaluate the stability of point and probabilistic forecasts across several retraining scenarios using three large forecastingdatasets and ten different global forecasting models. To analyze stability in the probabilistic setting, we propose a new model-agnostic, distribution-free, and scale-free metric that measuresprobabilistic stability: the Scaled Multi-Quantile Change (SMQC). The results show that less frequent retraining not only preserves but often improves forecast stability, challenging the need for frequent retraining. Moreover, the study shows that accuracy and stability are not necessarily conflicting objectives when adopting a global modeling approach. The study promotes a shift toward stability-aware forecasting practices, proposing a new metric to evaluate forecast stability effectively in probabilistic settings, and offering practical guidelines for building more stable and sustainable forecasting systems.