Forecaster's Dilemma: Extreme Events and Forecast Evaluation

TL;DR

This paper discusses the challenges of evaluating forecasts for extreme events, highlighting the forecaster's dilemma, and proposes proper weighted scoring rules as a solution, supported by theoretical, simulation, and real data analyses.

Contribution

It introduces the forecaster's dilemma, explains its implications, and advocates for proper weighted scoring rules to improve forecast evaluation for extreme events.

Findings

Conventional evaluation methods can discredit skillful forecasts for extreme events.

Proper weighted scoring rules can address the forecaster's dilemma.

Empirical analysis on U.S. inflation and GDP forecasts supports the proposed approach.

Abstract

In public discussions of the quality of forecasts, attention typically focuses on the predictive performance in cases of extreme events. However, the restriction of conventional forecast evaluation methods to subsets of extreme observations has unexpected and undesired effects, and is bound to discredit skillful forecasts when the signal-to-noise ratio in the data generating process is low. Conditioning on outcomes is incompatible with the theoretical assumptions of established forecast evaluation methods, thereby confronting forecasters with what we refer to as the forecaster's dilemma. For probabilistic forecasts, proper weighted scoring rules have been proposed as decision theoretically justifiable alternatives for forecast evaluation with an emphasis on extreme events. Using theoretical arguments, simulation experiments, and a real data study on probabilistic forecasts of U.S. inflation and gross domestic product growth, we illustrate and discuss the forecaster's dilemma along with potential remedies.