Fair Box ordinate transform for forecasts following a multivariate Gaussian law

TL;DR

This paper introduces a fair, dimension- and sample size-dependent Gaussian Box ordinate transform (BOT) to reliably assess calibration of multivariate Gaussian forecasts, especially with moderate sample sizes, demonstrated through simulations and weather forecast applications.

Contribution

The paper proposes a novel, fair version of the Gaussian sample BOT that improves calibration assessment accuracy for moderate sample sizes in multivariate Gaussian predictions.

Findings

Fair Gaussian sample BOT accurately detects miscalibration at moderate sample sizes.

Simulation results show improved reliability over alternative BOT versions.

Application to weather forecasts demonstrates practical utility.

Abstract

Monte Carlo techniques are the method of choice for making probabilistic predictions of an outcome in several disciplines. Usually, the aim is to generate calibrated predictions which are statistically indistinguishable from the outcome. Developers and users of such Monte Carlo predictions are interested in evaluating the degree of calibration of the forecasts. Here, we consider predictions of $p$-dimensional outcomes sampling a multivariate Gaussian distribution and apply the Box ordinate transform (BOT) to assess calibration. However, this approach is known to fail to reliably indicate calibration when the sample size n is moderate. For some applications, the cost of obtaining Monte-Carlo estimates is significant, which can limit the sample size, for instance, in model development when the model is improved iteratively. Thus, it would be beneficial to be able to reliably assess calibration even if the sample size n is moderate. To address this need, we introduce a fair, sample size- and dimension-dependent version of the Gaussian sample BOT. In a simulation study, the fair Gaussian sample BOT is compared with alternative BOT versions for different miscalibrations and for different sample sizes. Results confirm that the fair Gaussian sample BOT is capable of correctly identifying miscalibration when the sample size is moderate in contrast to the alternative BOT versions. Subsequently, the fair Gaussian sample BOT is applied to two to 12-dimensional predictions of temperature and vector wind using operational ensemble forecasts of the European Centre for Medium-Range Weather Forecasts (ECMWF). Firstly, perfectly reliable situations are considered where the outcome is replaced by a forecast that samples the same distribution as the members in the ensemble. Secondly, the BOT is computed using estimates of the actual temperature and vector wind from ECMWF analyses.