Proper Scoring Rules and Bregman Divergences

TL;DR

This paper unifies the mathematical foundations of proper scoring rules and Bregman divergences, emphasizing their geometric and convex analysis properties, and explores conditions for their uniqueness and differentiability.

Contribution

It introduces a unified theoretical framework linking proper scoring rules and Bregman divergences via convex extensions and subgradients, enhancing understanding of their geometric and differentiability properties.

Findings

Proper scoring rules can be derived from convex entropy extensions as subgradients.

Bregman divergences correspond to affine extensions of entropy functions.

Differentiability of entropy functions is characterized using directional derivatives and quasi-interior concepts.

Abstract

We revisit the mathematical foundations of proper scoring rules (PSRs) and Bregman divergences and present their characteristic properties in a unified theoretical framework. In many situations it is preferable not to generate a PSR directly from its convex entropy on the unit simplex but instead by the sublinear extension of the entropy to the positive orthant. This gives the scoring rule simply as a subgradient of the extended entropy, allowing for a more elegant theory. The other convex extensions of the entropy generate affine extensions of the scoring rule and induce the class of functional Bregman divergences. We discuss the geometric nature of the relationship between PSRs and Bregman divergences and extend and unify existing partial results. We also approach the topic of differentiability of entropy functions. Not all entropies of interest possess functional derivatives, but they do all have directional derivatives in almost every direction. Relying on the notion of quasi-interior of a convex set to quantify the latter property, we formalise under what conditions a PSR may be considered to be uniquely determined from its entropy.