Evaluating Forecasts with scoringutils in R

TL;DR

scoringutils is an R package that simplifies the evaluation and comparison of probabilistic forecasts across multiple dimensions, supporting complex forecast formats like predictive quantiles, with extensive visualization and diagnostic tools.

Contribution

scoringutils introduces a flexible, data.table-based framework for evaluating probabilistic forecasts, including support for predictive quantiles, and offers comprehensive tools for diagnostics, visualization, and score aggregation.

Findings

Supports evaluation of forecasts for COVID-19 cases and deaths

Provides extensive visualization and diagnostic tools

Extensible framework for custom scoring rules

Abstract

Evaluating forecasts is essential to understand and improve forecasting and make forecasts useful to decision makers. A variety of R packages provide a broad variety of scoring rules, visualisations and diagnostic tools. One particular challenge, which scoringutils aims to address, is handling the complexity of evaluating and comparing forecasts from several forecasters across multiple dimensions such as time, space, and different types of targets. scoringutils extends the existing landscape by offering a convenient and flexible data.table-based framework for evaluating and comparing probabilistic forecasts (forecasts represented by a full predictive distribution). Notably, scoringutils is the first package to offer extensive support for probabilistic forecasts in the form of predictive quantiles, a format that is currently used by several infectious disease Forecast Hubs. The package is easily extendable, meaning that users can supply their own scoring rules or extend existing classes to handle new types of forecasts. scoringutils provides broad functionality to check the data and diagnose issues, to visualise forecasts and missing data, to transform data before scoring, to handle missing forecasts, to aggregate scores, and to visualise the results of the evaluation. The paper presents the package and its core functionality and illustrates common workflows using example data of forecasts for COVID-19 cases and deaths submitted to the European COVID-19 Forecast Hub.