Optimal probabilistic forecasts: When do they work?

TL;DR

This paper investigates when probabilistic forecasts optimized under proper scoring rules outperform alternatives, emphasizing the importance of model compatibility and the effects of misspecification on forecast accuracy.

Contribution

It provides a detailed analysis of the conditions under which optimal probabilistic forecasts are superior, especially under model misspecification, using both theoretical insights and empirical illustrations.

Findings

Optimal forecasts outperform others when the model aligns well with the true process.

Superiority increases with greater model misspecification, given compatibility.

Forecast performance depends critically on the interplay between the true data process, model, and scoring rule.

Abstract

Proper scoring rules are used to assess the out-of-sample accuracy of probabilistic forecasts, with different scoring rules rewarding distinct aspects of forecast performance. Herein, we re-investigate the practice of using proper scoring rules to produce probabilistic forecasts that are `optimal' according to a given score, and assess when their out-of-sample accuracy is superior to alternative forecasts, according to that score. Particular attention is paid to relative predictive performance under misspecification of the predictive model. Using numerical illustrations, we document several novel findings within this paradigm that highlight the important interplay between the true data generating process, the assumed predictive model and the scoring rule. Notably, we show that only when a predictive model is sufficiently compatible with the true process to allow a particular score criterion to reward what it is designed to reward, will this approach to forecasting reap benefits. Subject to this compatibility however, the superiority of the optimal forecast will be greater, the greater is the degree of misspecification. We explore these issues under a range of different scenarios, and using both artificially simulated and empirical data.