Proper Scoring and Sufficiency

TL;DR

This paper explores the role of Bregman divergences in various fields like information theory and statistics, highlighting how sufficiency conditions influence their relationships and transferability of results.

Contribution

It demonstrates the connection between Bregman divergences and information divergence under sufficiency conditions across multiple disciplines.

Findings

Bregman divergences are central to optimization in multiple fields.

Sufficiency conditions determine when Bregman divergence equals information divergence.

Differences in sufficiency conditions explain transferability of results across disciplines.

Abstract

Logarithmic score and information divergence appear in both information theory, statistics, statistical mechanics, and portfolio theory. We demonstrate that all these topics involve some kind of optimization that leads directly to the use of Bregman divergences. If a sufficiency condition is also fulfilled the Bregman divergence must be proportional to information divergence. The sufficiency condition has quite different consequences in the different areas of application, and often it is not fulfilled. Therefore the sufficiency condition can be used to explain when results from one area can be transferred directly from one area to another and when one will experience differences.