Potential root mean square error skill score

TL;DR

This paper introduces the potential RMSE skill score, a new verification metric for solar forecasts that accounts for consistency between forecast optimization strategies and verification measures, based on correlations.

Contribution

It proposes a simplified, correlation-based RMSE skill score that reduces discrimination against forecasts optimized under different strategies.

Findings

The potential RMSE skill score depends only on forecast-observation crosscorrelation and observation autocorrelation.

It simplifies calculation while maintaining fairness across different forecast optimization strategies.

The score allows MAE-optimized forecasts to achieve high RMSE skill scores.

Abstract

Consistency, in a narrow sense, denotes the alignment between the forecast-optimization strategy and the verification directive. The current recommended deterministic solar forecast verification practice is to report the skill score based on root mean square error (RMSE), which would violate the notion of consistency if the forecasts are optimized under another strategy such as minimizing the mean absolute error (MAE). This paper overcomes such difficulty by proposing a so-called "potential RMSE skill score," which depends only on: (1) the crosscorrelation between forecasts and observations, and (2) the autocorrelation of observations. While greatly simplifying the calculation, the new skill score does not discriminate inconsistent forecasts as much, e.g., even MAE-optimized forecasts can attain a high RMSE skill score.