Sequential Scoring Rule Evaluation for Forecast Method Selection

TL;DR

This paper develops a sequential statistical framework for selecting forecasting methods using scoring rules, introducing new theoretical results and demonstrating improved testing power through simulations.

Contribution

It introduces a novel large deviations analysis for sequential scoring rule evaluation and integrates it with generalized e-values for more powerful, reliable method selection.

Findings

SSRE terminates in finite time with probability one

The approach outperforms traditional testing methods in simulated scenarios

Moments of the stopping time are proven to exist

Abstract

This paper shows that sequential statistical analysis techniques can be generalised to the problem of selecting between alternative forecasting methods using scoring rules. A return to basic principles is necessary in order to show that ideas and concepts from sequential statistical methods can be adapted and applied to sequential scoring rule evaluation (SSRE). One key technical contribution of this paper is the development of a large deviations type result for SSRE schemes using a change of measure that parallels a traditional exponential tilting form. Further, we also show that SSRE will terminate in finite time with probability one, and that the moments of the SSRE stopping time exist. A second key contribution is to show that the exponential tilting form underlying our large deviations result allows us to cast SSRE within the framework of generalised e-values. Relying on this formulation, we devise sequential testing approaches that are both powerful and maintain control on error probabilities underlying the analysis. Through several simulated examples, we demonstrate that our e-values based SSRE approach delivers reliable results that are more powerful than more commonly applied testing methods precisely in the situations where these commonly applied methods can be expected to fail.