Random Noise vs State-of-the-Art Probabilistic Forecasting Methods : A Case Study on CRPS-Sum Discrimination Ability

TL;DR

This paper critically evaluates the CRPS-sum metric for multivariate probabilistic forecasting, revealing its limitations and potential to misjudge model performance, especially when comparing complex models to simple noise-based ones.

Contribution

It systematically analyzes CRPS-sum's discrimination ability, identifies its flaws, and demonstrates how it can misleadingly favor random noise models over advanced methods.

Findings

CRPS-sum's performance depends on data properties.

CRPS-sum overlooks per-dimension performance.

Dummy noise models can outperform complex models under CRPS-sum.

Abstract

The recent developments in the machine learning domain have enabled the development of complex multivariate probabilistic forecasting models. Therefore, it is pivotal to have a precise evaluation method to gauge the performance and predictability power of these complex methods. To do so, several evaluation metrics have been proposed in the past (such as Energy Score, Dawid-Sebastiani score, variogram score), however, they cannot reliably measure the performance of a probabilistic forecaster. Recently, CRPS-sum has gained a lot of prominence as a reliable metric for multivariate probabilistic forecasting. This paper presents a systematic evaluation of CRPS-sum to understand its discrimination ability. We show that the statistical properties of target data affect the discrimination ability of CRPS-Sum. Furthermore, we highlight that CRPS-Sum calculation overlooks the performance of the model on each dimension. These flaws can lead us to an incorrect assessment of model performance. Finally, with experiments on the real-world dataset, we demonstrate that the shortcomings of CRPS-Sum provide a misleading indication of the probabilistic forecasting performance method. We show that it is easily possible to have a better CRPS-Sum for a dummy model, which looks like random noise, in comparison to the state-of-the-art method.