Uncertainty Quantification in Forecast Comparisons

TL;DR

This paper introduces a statistically rigorous framework for quantifying uncertainty in forecast evaluation across multiple dimensions, addressing limitations of existing methods.

Contribution

It develops simultaneous confidence bands for scores and skill scores, enabling joint inference in complex, multivariate forecast comparisons.

Findings

Bootstrap-based confidence bands are valid under multivariate Diebold-Mariano assumptions.

The framework effectively quantifies benefits of time-varying models in macroeconomics.

It compares data-driven and physics-based weather models using joint uncertainty measures.

Abstract

Skill scores, which measure the relative improvement of a forecasting method over a benchmark via consistent scoring functions and proper scoring rules, are a standard tool in forecast evaluation, yet their sampling uncertainty is rarely rigorously quantified. With modern forecasting applications being increasingly multivariate and involving evaluations across multiple horizons, variables, spatial locations, and forecasting methods, standard tools like the pairwise Diebold-Mariano forecast accuracy test or pointwise confidence intervals fail to account for the multiple comparison problem, leading to inflated Type I error rates and invalid joint inference. To address the lack of a coherent, statistically rigorous framework for quantifying uncertainty across these multi-dimensional evaluation problems, we introduce simultaneous confidence bands for expected scores and skill scores. Our framework provides a versatile tool for joint inference that is applicable to any forecast type from mean and quantile to full distributional forecasts. We develop a bootstrap implementation and show that our bands are valid under multivariate extensions of the classical Diebold-Mariano assumptions. We demonstrate the practical utility of the approach in two case studies by quantifying the benefits of time-varying parameter models for macroeconomic forecasting, and by comparing data-driven and physics-based models in probabilistic weather forecasting.