Reduced Perplexity: A simplified perspective on assessing probabilistic forecasts

TL;DR

This paper introduces a simplified method for evaluating probabilistic forecasts by translating Shannon information metrics into the probability domain, revealing equivalences and new insights into forecast accuracy and robustness.

Contribution

It provides a novel perspective by connecting information theory metrics with probabilistic forecast assessment, introducing the concept of a Risk Profile spectrum.

Findings

Negative logarithmic score equals geometric mean of probabilities.

Reduced perplexity indicates better probabilistic inference.

Generalized means form a spectrum of performance metrics.

Abstract

A simple, intuitive approach to the assessment of probabilistic inferences is introduced. The Shannon information metrics are translated to the probability domain. The translation shows that the negative logarithmic score and the geometric mean are equivalent measures of the accuracy of a probabilistic inference. Thus there is both a quantitative reduction in perplexity, which is the inverse of the geometric mean of the probabilities, as good inference algorithms reduce the uncertainty and a qualitative reduction due to the increased clarity between the original set of probabilistic forecasts and their central tendency, the geometric mean. Further insight is provided by showing that the R\'enyi and Tsallis entropy functions translated to the probability domain are both the weighted generalized mean of the distribution. The generalized mean of probabilistic forecasts forms a spectrum of performance metrics referred to as a Risk Profile. The arithmetic mean is used to measure the decisiveness, while the -2/3 mean is used to measure the robustness.