Scoring Alternative Forecast Distributions: Completing the Kullback Distance Complex

TL;DR

This paper introduces a novel framework for evaluating and comparing forecast distributions using a Pareto optimal exchange of scoring outcomes, extending the Kullback-Leibler divergence into a comprehensive four-dimensional 'Kullback complex' that captures complete information content.

Contribution

It develops a new scoring exchange method that avoids arbitrariness and extends the Kullback divergence into a four-component measure called the 'Kullback complex' for complete information characterization.

Findings

Pareto optimal scoring exchange improves forecast comparison

Kullback complex provides a complete information measure

Extension of Kullback divergence with three additional components

Abstract

We develop two surprising new results regarding the use of proper scoring rules for evaluating the predictive quality of two alternative sequential forecast distributions. Both of the proponents prefer to be awarded a score derived from the other's distribution rather than a score awarded on the basis of their own. A Pareto optimal exchange of their scoring outcomes provides the basis for a comparison of forecast quality that is preferred by both forecasters, and also evades a feature of arbitrariness inherent in using the forecasters' own achieved scores. The well-known Kullback divergence, used as a measure of information, is evaluated via the entropies in the two forecast distributions and the two cross-entropies between them. We show that Kullback's symmetric measure needs to be appended by three component measures if it is to characterise completely the information content of the two asserted probability forecasts. Two of these do not involve entropies at all. The resulting 'Kullback complex' supported by the 4-dimensional measure is isomorphic to an equivalent vector measure generated by the forecasters' expectations of their scores, each for one's own score and for the other's score. We foreshadow the results of a sophisticated application of the Pareto relative scoring procedure for actual sequentional observations, and we propose a standard format for evaluation.