A scoring framework for tiered warnings and multicategorical forecasts based on fixed risk measures

TL;DR

This paper introduces a flexible scoring framework for evaluating tiered warnings and multicategorical forecasts in meteorology, aligning scores with fixed risk measures and user-specific thresholds for improved forecast assessment.

Contribution

It presents a novel family of scoring functions that incorporate fixed risk parameters and use-case specific weights, offering an alternative to existing performance measures.

Findings

Scores are consistent with fixed threshold probabilities.

Framework rewards accurate discrimination between categories.

Discounted penalties for near misses improve forecast evaluation.

Abstract

The use of tiered warnings and multicategorical forecasts are ubiquitous in meteorological operations. Here, a flexible family of scoring functions is presented for evaluating the performance of ordered multicategorical forecasts. Each score has a risk parameter $\alpha$, selected for the specific use case, so that it is consistent with a forecast directive based on the fixed threshold probability $1-\alpha$ (equivalently, a fixed $\alpha$-quantile mapping). Each score also has use-case specific weights so that forecasters who accurately discriminate between categorical thresholds are rewarded in proportion to the weight for that threshold. A variation is presented where the penalty assigned to near misses or close false alarms is discounted, which again is consistent with directives based on fixed risk measures. The scores presented provide an alternative to many performance measures currently in use, whose optimal threshold probabilities for forecasting an event typically vary with each forecast case, and in the case of equitable scores are based around sample base rates rather than risk measures suitable for users.