Evaluating probability forecasts

TL;DR

This paper develops a new statistical framework for evaluating probability forecasts using loss functions and martingale theory, providing a more rigorous approach to assess forecast accuracy over time.

Contribution

It introduces an alternative evaluation method for probability forecasts that leverages loss functions and martingale theory, enhancing the assessment process.

Findings

Provides a theoretical foundation for scoring rules in probability forecasting.

Proposes a martingale-based approach to evaluate forecast accuracy.

Offers a rigorous method applicable to climate, finance, and medical predictions.

Abstract

Probability forecasts of events are routinely used in climate predictions, in forecasting default probabilities on bank loans or in estimating the probability of a patient's positive response to treatment. Scoring rules have long been used to assess the efficacy of the forecast probabilities after observing the occurrence, or nonoccurrence, of the predicted events. We develop herein a statistical theory for scoring rules and propose an alternative approach to the evaluation of probability forecasts. This approach uses loss functions relating the predicted to the actual probabilities of the events and applies martingale theory to exploit the temporal structure between the forecast and the subsequent occurrence or nonoccurrence of the event.