Scoring Rules and Calibration for Imprecise Probabilities

TL;DR

This paper extends the concepts of proper scoring rules and calibration from precise to imprecise probabilistic forecasts, providing a theoretical framework and practical insights for decision-making under uncertainty.

Contribution

It generalizes scoring rules and calibration for imprecise probabilities, linking them to distributional robustness and highlighting their distinct roles.

Findings

Proper scoring rules and calibration are not necessarily aligned in the imprecise case.

The concept of decision-theoretic entropy is central to both scoring and calibration.

Illustrates pitfalls in loss function choices in machine learning distributional robustness.

Abstract

What does it mean to say that, for example, the probability for rain tomorrow is between 20% and 30%? The theory for the evaluation of precise probabilistic forecasts is well-developed and is grounded in the key concepts of proper scoring rules and calibration. For the case of imprecise probabilistic forecasts (sets of probabilities), such theory is still lacking. In this work, we therefore generalize proper scoring rules and calibration to the imprecise case. We develop these concepts as relative to data models and decision problems. As a consequence, the imprecision is embedded in a clear context. We establish a close link to the paradigm of (group) distributional robustness and in doing so provide new insights for it. We argue that proper scoring rules and calibration serve two distinct goals, which are aligned in the precise case, but intriguingly are not necessarily aligned in the imprecise case. The concept of decision-theoretic entropy plays a key role for both goals. Finally, we demonstrate the theoretical insights in machine learning practice, in particular we illustrate subtle pitfalls relating to the choice of loss function in distributional robustness.