Evaluating ensemble forecasts by the Ignorance score -- Correcting the finite-ensemble bias

TL;DR

This paper introduces a bias-corrected Ignorance score for ensemble forecasts, ensuring fairer evaluation across different ensemble sizes by removing size-related bias and improving accuracy.

Contribution

A new unbiased estimator of the Ignorance score is derived, correcting finite-ensemble bias and enabling fair comparison of ensemble forecasts regardless of size.

Findings

Bias in standard Ignorance score depends on ensemble size

The new estimator reduces bias and variance in score evaluation

Bias correction improves ranking of physical ensembles over climatological ones

Abstract

This study considers the application of the Ignorance Score (also known as the Logarithmic Score) in the context of ensemble verification. In particular, we consider the case where an ensemble forecast is transformed to a Normal forecast distribution, and this distribution is evaluated by the Ignorance Score. It is shown that the standard Ignorance score is biased with respect to the ensemble size, such that larger ensembles yield systematically better expected scores. A new estimator of the Ignorance score is derived which is unbiased with respect to the ensemble size. In an application to seasonal climate predictions it is shown that the standard Ignorance score assigns better expected scores to simple climatological ensembles or biased ensembles that have many members, than to physical dynamical and unbiased ensembles with fewer members. By contrast, the new bias-corrected Ignorance score ranks the physical dynamical and unbiased ensembles better than the climatological and biased ones, independent of ensemble size. It is shown that the unbiased estimator has smaller estimator variance and error than the standard estimator, and that it is a fair verification score, which is optimized if the ensemble members are statistically consistent with the observations. The finite ensemble bias of ensemble verification scores is discussed more broadly. It is argued that a bias-correction is appropriate when forecast systems with different ensemble sizes are compared, and when an evaluation of the underlying distribution of the ensemble is of interest; possible applications to unbiased parameter estimation are discussed.