Quantile forecast discrimination ability and value

TL;DR

This paper introduces new tools for assessing the discrimination ability and economic value of quantile forecasts in continuous variables, extending forecast verification methods beyond categorical data.

Contribution

It develops the RUC curve and quantile value plot as novel, user-focused tools for evaluating forecast discrimination and value in a nonparametric framework.

Findings

RUC curve effectively measures forecast discrimination for specific users.

Quantile value plot links discrimination ability to economic value.

Application to real and synthetic data demonstrates utility.

Abstract

While probabilistic forecast verification for categorical forecasts is well established, some of the existing concepts and methods have not found their equivalent for the case of continuous variables. New tools dedicated to the assessment of forecast discrimination ability and forecast value are introduced here, based on quantile forecasts being the base product for the continuous case (hence in a nonparametric framework). The relative user characteristic (RUC) curve and the quantile value plot allow analysing the performance of a forecast for a specific user in a decision-making framework. The RUC curve is designed as a user-based discrimination tool and the quantile value plot translates forecast discrimination ability in terms of economic value. The relationship between the overall value of a quantile forecast and the respective quantile skill score is also discussed. The application of these new verification approaches and tools is illustrated based on synthetic datasets, as well as for the case of global radiation forecasts from the high resolution ensemble COSMO-DE-EPS of the German Weather Service.