How to derive skill from the Fractions Skill Score

TL;DR

This paper introduces a new method to interpret the Fractions Skill Score (FSS) by deriving a meaningful reference score based on random forecasts, improving the assessment of forecast skill across different length scales.

Contribution

It provides a novel approach to determine forecast skill from FSS by deriving an expression for random forecasts, enhancing interpretability and robustness.

Findings

The new method significantly alters the length scales considered skillful.

It reveals subtleties in FSS interpretation previously overlooked.

The approach offers a more meaningful benchmark for forecast skill assessment.

Abstract

The Fractions Skill Score (FSS) is a widely used metric for assessing forecast skill, with applications ranging from precipitation to volcanic ash forecasts. By evaluating the fraction of grid squares exceeding a threshold in a neighbourhood, the intuition is that it can avoid the pitfalls of pixel-wise comparisons and identify length scales at which a forecast has skill. The FSS is typically interpreted relative to a `useful' criterion, where a forecast is considered skillful if its score exceeds a simple reference score. However, the typical reference score used is problematic, as it is not derived in a way that provides obvious meaning or that scales with neighbourhood size, and forecasts that do not exceed it can have considerable skill. We therefore provide a new method to determine forecast skill from the FSS, by deriving an expression for the FSS achieved by a random forecast, which provides a more robust and meaningful reference score to compare with. Through illustrative examples we show that this new method considerably changes the length scales at which a forecast would be regarded as skillful, and reveals subtleties in how the FSS should be interpreted.