Scoring rule nets: beyond mean target prediction in multivariate regression

TL;DR

This paper introduces Conditional CRPS, a new multivariate scoring rule that improves probabilistic regression models by better capturing correlations and outperforming traditional MLE in various datasets.

Contribution

The paper proposes Conditional CRPS, a novel multivariate scoring rule with closed-form expressions, enhancing model calibration and correlation sensitivity beyond existing methods.

Findings

Conditional CRPS outperforms MLE in experiments.

It is comparable to state-of-the-art non-parametric models.

The scoring rule effectively captures correlations in multivariate data.

Abstract

Probabilistic regression models trained with maximum likelihood estimation (MLE), can sometimes overestimate variance to an unacceptable degree. This is mostly problematic in the multivariate domain. While univariate models often optimize the popular Continuous Ranked Probability Score (CRPS), in the multivariate domain, no such alternative to MLE has yet been widely accepted. The Energy Score - the most investigated alternative - notoriously lacks closed-form expressions and sensitivity to the correlation between target variables. In this paper, we propose Conditional CRPS: a multivariate strictly proper scoring rule that extends CRPS. We show that closed-form expressions exist for popular distributions and illustrate their sensitivity to correlation. We then show in a variety of experiments on both synthetic and real data, that Conditional CRPS often outperforms MLE, and produces results comparable to state-of-the-art non-parametric models, such as Distributional Random Forest (DRF).