Contrasting Probabilistic Scoring Rules

TL;DR

This paper compares various proper scoring rules for probabilistic forecasts, highlighting their differences in how they favor uncertainty levels, and offers guidance on selecting appropriate scoring rules.

Contribution

It provides a comparative analysis of common proper scoring rules and clarifies their preferences regarding uncertainty in probabilistic forecasting.

Findings

Logarithmic scoring favors higher uncertainty.

Spherical scoring prefers lower uncertainty.

Other scoring rules are indifferent to uncertainty levels.

Abstract

There are several scoring rules that one can choose from in order to score probabilistic forecasting models or estimate model parameters. Whilst it is generally agreed that proper scoring rules are preferable, there is no clear criterion for preferring one proper scoring rule above another. This manuscript compares and contrasts some commonly used proper scoring rules and provides guidance on scoring rule selection. In particular, it is shown that the logarithmic scoring rule prefers erring with more uncertainty, the spherical scoring rule prefers erring with lower uncertainty, whereas the other scoring rules are indifferent to either option.