# Evaluating probabilistic forecasts of extremes using continuous ranked probability score distributions

## TL;DR

This paper investigates the effectiveness of the continuous ranked probability score (CRPS) in evaluating probabilistic forecasts of extreme events, proposing a new approach based on extreme value theory for better assessment.

## Contribution

It introduces a formal framework for evaluating extreme event forecasts and proposes a novel method using extreme value theory to improve assessment accuracy.

## Key findings

- CRPS is not suitable for extreme event verification when assessed by expectation.
- A new index based on extreme value theory effectively compares calibrated forecasts for extremes.
- The proposed method's strengths and limitations are analyzed through theory and simulations.

## Abstract

Verifying probabilistic forecasts for extreme events is a highly active research area because popular media and public opinions are naturally focused on extreme events, and biased conclusions are readily made. In this context, classical verification methods tailored for extreme events, such as thresholded and weighted scoring rules, have undesirable properties that cannot be mitigated, and the well-known continuous ranked probability score (CRPS) is no exception. In this paper, we define a formal framework for assessing the behavior of forecast evaluation procedures with respect to extreme events, which we use to demonstrate that assessment based on the expectation of a proper score is not suitable for extremes. Alternatively, we propose studying the properties of the CRPS as a random variable by using extreme value theory to address extreme event verification. An index is introduced to compare calibrated forecasts, which summarizes the ability of probabilistic forecasts for predicting extremes. The strengths and limitations of this method are discussed using both theoretical arguments and simulations.