Ensemble size dependence of the logarithmic score for forecasts issued as multivariate normal distributions

TL;DR

This paper derives a relationship describing how the logarithmic score for multivariate normal ensemble forecasts depends on ensemble size, enabling fair comparison across different ensemble sizes and improving forecast verification.

Contribution

It generalizes the 2018 univariate ensemble size adjustment to multivariate normal distributions, allowing for more accurate forecast evaluation with varying ensemble sizes.

Findings

Fair logarithmic scores are nearly independent of ensemble size after adjustment.

Unadjusted scores decrease with increasing ensemble size, highlighting the need for correction.

Application to ECMWF forecasts demonstrates practical usefulness of the adjustment.

Abstract

Multivariate probabilistic verification is concerned with the evaluation of joint probability distributions of vector quantities such as a weather variable at multiple locations or a wind vector for instance. The logarithmic score is a proper score that is useful in this context. In order to apply this score to ensemble forecasts, a choice for the density is required. Here, we are interested in the specific case when the density is multivariate normal with mean and covariance given by the ensemble mean and ensemble covariance, respectively. Under the assumptions of multivariate normality and exchangeability of the ensemble members, a relationship is derived which describes how the logarithmic score depends on ensemble size. It permits to estimate the score in the limit of infinite ensemble size from a small ensemble and thus produces a fair logarithmic score for multivariate ensemble forecasts under the assumption of normality. This generalises a study from 2018 which derived the ensemble size adjustment of the logarithmic score in the univariate case. An application to medium-range forecasts examines the usefulness of the ensemble size adjustments when multivariate normality is only an approximation. Predictions of vectors consisting of several different combinations of upper air variables are considered. Logarithmic scores are calculated for these vectors using ECMWF's daily extended-range forecasts which consist of a 100-member ensemble. The probabilistic forecasts of these vectors are verified against operational ECMWF analyses in the Northern mid-latitudes in autumn 2023. Scores are computed for ensemble sizes from 8 to 100. The fair logarithmic scores of ensembles with different cardinalities are very close, in contrast to the unadjusted scores which decrease considerably with ensemble size. This provides evidence for the practical usefulness of the derived relationships.