Unbiased calculation, evaluation, and calibration of ensemble forecast anomalies

TL;DR

This paper addresses the statistical inconsistencies in calculating ensemble forecast anomalies, proposing methods to ensure unbiased variance and spread-error ratios, thereby improving reliability and comparability of probabilistic forecasts.

Contribution

It introduces estimators and approaches for anomaly calculation that are statistically consistent and unbiased, enhancing forecast evaluation accuracy.

Findings

Unbiased variance and spread-error ratios can be achieved with new estimators.

Constructing member-specific climatologies ensures consistent anomalies.

Alternative anomaly calculation methods influence probabilistic forecast skill.

Abstract

Long-range ensemble forecasts are typically verified as anomalies with respect to a lead-time dependent climatological mean to remove the influence of systematic biases. However, common methods for calculating anomalies result in statistical inconsistencies between forecast and verification anomalies, even for a perfectly reliable ensemble. It is important to account for these systematic effects when evaluating ensemble forecast systems, particularly when tuning a model to improve the reliability of forecast anomalies or when comparing spread-error diagnostics between systems with different reforecast periods. Here, we show that unbiased variances and spread-error ratios can be recovered by deriving estimators that are consistent with the values that would be achieved when calculating anomalies relative to the true, but unknown, climatological mean. An elegant alternative is to construct forecast climatologies separately for each member, which ensures that forecast and verification anomalies are defined relative to reference climatological means with the same sampling uncertainty. This alternative approach has no impact on forecast ensemble means but systematically modifies the total variance and ensemble spread of forecast anomalies in such a way that anomaly-based spread-error ratios are unbiased without any explicit correction for climatology sample size. Furthermore, the improved statistical consistency of forecast and verification anomalies means that probabilistic forecast skill is optimised when the underlying forecast is also perfectly reliable. Alternative methods for anomaly calculation can thus impact probabilistic forecast skill, especially when anomalies are defined relative to climatologies with a small sample size. Finally, we demonstrate the equivalence of anomalies calculated using different methods after applying an unbiased statistical calibration.