Cross-calibration of probabilistic forecasts

TL;DR

This paper extends the concept of calibration in probabilistic forecasting to cross-calibration among multiple forecasters, providing theoretical insights, diagnostic tools, and practical applications, including inflation rate forecasts by the Bank of England.

Contribution

It introduces the notion of cross-calibration, develops diagnostic tools and tests for it, and demonstrates its importance through simulations and real-world inflation forecast data.

Findings

Cross-calibration is a stronger condition than calibration.

Methods effectively assess cross-calibration in simulated data.

Application to inflation forecasts shows practical utility.

Abstract

When providing probabilistic forecasts for uncertain future events, it is common to strive for calibrated forecasts, that is, the predictive distribution should be compatible with the observed outcomes. Several notions of calibration are available in the case of a single forecaster alongside with diagnostic tools and statistical tests to assess calibration in practice. Often, there is more than one forecaster providing predictions, and these forecasters may use information of the others and therefore influence one another. We extend common notions of calibration, where each forecaster is analysed individually, to notions of cross-calibration where each forecaster is analysed with respect to the other forecasters in a natural way. It is shown theoretically and in simulation studies that cross-calibration is a stronger requirement on a forecaster than calibration. Analogously to calibration for individual forecasters, we provide diagnostic tools and statistical tests to assess forecasters in terms of cross-calibration. The methods are illustrated in simulation examples and applied to probabilistic forecasts for inflation rates by the Bank of England.