Tail calibration of probabilistic forecasts

TL;DR

This paper introduces a new framework for evaluating the reliability of probabilistic forecasts specifically for extreme outcomes, addressing a gap in existing calibration methods and providing practical diagnostic tools.

Contribution

It proposes a general notion of tail calibration for probabilistic forecasts, linking it to existing calibration concepts and extreme value theory, with practical diagnostic tools and a case study.

Findings

Tail calibration effectively assesses extreme outcome reliability.

Diagnostic tools improve evaluation of probabilistic forecasts for tails.

Case study demonstrates practical application in weather forecasting.

Abstract

Probabilistic forecasts comprehensively describe the uncertainty in the unknown future outcome, making them essential for decision making and risk management. While several methods have been introduced to evaluate probabilistic forecasts, existing evaluation techniques are ill-suited to the evaluation of tail properties of such forecasts. However, these tail properties are often of particular interest to forecast users due to the severe impacts caused by extreme outcomes. In this work, we introduce a general notion of tail calibration for probabilistic forecasts, which allows forecasters to assess the reliability of their predictions for extreme outcomes. We study the relationships between tail calibration and standard notions of forecast calibration, and discuss connections to peaks-over-threshold models in extreme value theory. Diagnostic tools are introduced and applied in a case study on European precipitation forecasts