Investigating How Neighbourhood Scores Reflect Forecast Error

TL;DR

This paper theoretically examines how two spatial forecast verification scores, FSS and BDnSS, reflect forecast errors and their tendencies to favor certain forecast characteristics, revealing differences in their sensitivity to errors and smoothing.

Contribution

It provides a theoretical analysis of how FSS and BDnSS scores rank forecasts with different errors, highlighting their biases and sensitivities in spatial verification.

Findings

FSS tends to favor over-predicting mean frequency, especially with percentile thresholds.

Both scores favor smoother forecasts, illustrating the double penalty problem.

BDnSS is more affected by the double penalty issue than FSS.

Abstract

Meaningful scores for forecast verification are essential for developing reliable forecasts, and there has been much effort to develop scores that align well with human perceptions of forecast quality. Whilst many of these scores have intuitive interpretations, relatively little is known about how these scores rank different forecasts, and how scores reflect forecast error. We theoretically explore the behaviour of two scores that fall within the `neighbourhood' paradigm of spatial verification; the Fractions Skill Score (FSS) and Brier Divergence Skill Score (BDnSS). We investigate how each score ranks forecasts with two types of error; errors in the mean frequency (corresponding to intensity or shape errors) and errors in the standard deviation (corresponding to errors in spatial structure, such as blurring or excess noise). We find that under many situations the FSS assigns higher scores to forecasts that over-predict mean frequency, thus theoretically confirming the need to use the FSS with percentile thresholds. Both scores assign higher scores to smoother forecasts in many situations, a reflection of the `double penalty' problem; however, we observe that size of this effect is larger for the BDnSS than the FSS, showing that the FSS under some situations is less susceptible to the double penalty problem than the BDnSS.