A User-Focused Approach to Evaluating Probabilistic and Categorical Forecasts

TL;DR

This paper presents a user-centric evaluation framework for probabilistic binary forecasts, emphasizing decision thresholds, proper scoring rules, and Murphy diagrams to improve interpretability and comparison of forecast systems.

Contribution

It introduces a comprehensive, decision-focused evaluation approach using Murphy diagrams and the FIRM score, enhancing the assessment of probabilistic forecasts over traditional categorical methods.

Findings

Murphy diagrams offer better insight into forecast performance across decision thresholds.

Proper scoring rules can be tailored to user decision importance.

The FIRM score enables fair comparison between probabilistic and categorical forecasts.

Abstract

A user-focused verification approach for evaluating probability forecasts of binary outcomes (also known as probabilistic classifiers) is demonstrated that is (i) based on proper scoring rules, (ii) focuses on user decision thresholds, and (iii) provides actionable insights. It is argued that when categorical performance diagrams and the critical success index are used to evaluate overall predictive performance, rather than the discrimination ability of probabilistic forecasts, they may produce misleading results. Instead, Murphy diagrams are shown to provide better understanding of overall predictive performance as a function of user probabilistic decision threshold. It is illustrated how to select a proper scoring rule, based on the relative importance of different user decision thresholds, and how this choice impacts scores of overall predictive performance and supporting measures of discrimination and calibration. These approaches and ideas are demonstrated using several probabilistic thunderstorm forecast systems as well as synthetic forecast data. Furthermore, a fair method for comparing the performance of probabilistic and categorical forecasts is illustrated using the FIxed Risk Multicategorical (FIRM) score, which is a proper scoring rule directly connected to values on the Murphy diagram. While the methods are illustrated using thunderstorm forecasts, they are applicable for evaluating probabilistic forecasts for any situation with binary outcomes.