On Comparison Of Experts

TL;DR

This paper introduces a comparison test for evaluating two experts' probabilistic forecasts based on their predictions and actual outcomes, ensuring the test adheres to natural properties and is uniquely determined by the derivative of the induced measures.

Contribution

It proposes two natural properties for comparison tests and demonstrates that these properties uniquely determine a specific test based on measure derivatives.

Findings

The proposed test is uniquely characterized by natural properties.

The test relies on the derivative of the measures induced by the experts.

The approach provides a systematic way to rank experts based on forecast accuracy.

Abstract

A policy maker faces a sequence of unknown outcomes. At each stage two (self-proclaimed) experts provide probabilistic forecasts on the outcome in the next stage. A comparison test is a protocol for the policy maker to (eventually) decide which of the two experts is better informed. The protocol takes as input the sequence of pairs of forecasts and actual realizations and (weakly) ranks the two experts. We propose two natural properties that such a comparison test must adhere to and show that these essentially uniquely determine the comparison test. This test is a function of the derivative of the induced pair of measures at the realization.