Decompositions of the mean continuous ranked probability score

TL;DR

This paper introduces a new isotonicity-based decomposition of the mean continuous ranked probability score (CRPS) that effectively quantifies calibration, discrimination, and uncertainty in probabilistic forecasts, with theoretical analysis and practical case studies.

Contribution

It proposes a novel isotonicity-based decomposition of the mean CRPS, improving interpretability and guaranteeing nonnegativity of components, compared to previous empirical methods.

Findings

The new decomposition guarantees nonnegativity of components.

It provides a stronger quantification of calibration.

Case studies demonstrate practical applicability.

Abstract

The continuous ranked probability score (crps) is the most commonly used scoring rule in the evaluation of probabilistic forecasts for real-valued outcomes. To assess and rank forecasting methods, researchers compute the mean crps over given sets of forecast situations, based on the respective predictive distributions and outcomes. We propose a new, isotonicity-based decomposition of the mean crps into interpretable components that quantify miscalibration (MSC), discrimination ability (DSC), and uncertainty (UNC), respectively. In a detailed theoretical analysis, we compare the new approach to empirical decompositions proposed earlier, generalize to population versions, analyse their properties and relationships, and relate to a hierarchy of notions of calibration. The isotonicity-based decomposition guarantees the nonnegativity of the components and quantifies calibration in a sense that is stronger than for other types of decompositions, subject to the nondegeneracy of empirical decompositions. We illustrate the usage of the isotonicity-based decomposition in case studies from weather prediction and machine learning.