A generalisation of the signal-to-noise ratio using proper scoring rules

TL;DR

This paper introduces a generalized signal-to-noise ratio concept based on proper scoring rules, broadening its applicability to various forecast types and illustrating it with numerical examples including ensemble forecasts and NAO index predictions.

Contribution

It extends the classical RPC to a more general form using proper scoring rules, enabling analysis of diverse forecast types and resolving previous interpretative ambiguities.

Findings

RPC and scoring rule-based ratios agree on synthetic data.

Different statistics show varying variance, indicating different properties.

NAO data results are more ambiguous regarding anomalous ratios.

Abstract

A generalised concept of the signal-to-noise ratio (or equivalently the ratio of predictable components, or RPC) is provided, based on proper scoring rules. This definition is the natural generalisation of the classical RPC, yet it allows one to define and analyse the signal-to-noise properties of any type of forecast that is amenable to scoring, thus drastically widening the applicability of these concepts. The methodology is illustrated through numerical examples of ensemble forecasts, scored using the continuous ranked probability score (CRPS), and of probability forecasts of a binary event, scored using the logarithmic score. Numerical examples are carried out using both synthetic data with prescribed signal-to-noise ratios as well as seasonal ensemble hindcasts of the North Atlantic Oscillation (NAO) index. The latter have previously been interpreted as having a signal-to-noise "paradox", or anomalous signal-to-noise ratio, using the RPC statistic. For the synthetic data, the RPC statistic as well as the scoring rule-based ones agree regarding which data sets exhibit anomalous signal-to-noise ratios, but exhibit different variance, indicating different statistical properties. For the NAO data, on the other hand, the different statistics are more equivocal on whether the signal-to-noise ratio is anomalous.