A Cardinal Comparison of Experts

TL;DR

This paper introduces an axiomatic method for comparing and ranking experts based on their probabilistic forecasts over time, establishing a nearly unique test related to likelihood ratios that can reliably identify the better expert.

Contribution

It proposes a set of axioms for expert comparison tests, introduces a nearly unique test based on likelihood ratios, and proves its effectiveness in identifying the superior expert over finite time.

Findings

The proposed test satisfies natural axioms for expert comparison.

The test is essentially unique and closely related to likelihood ratios.

It can reliably detect the informed expert within finite time when experts' advice differ sufficiently.

Abstract

In various situations, decision makers face experts that may provide conflicting advice. This advice may be in the form of probabilistic forecasts over critical future events. We consider a setting where the two forecasters provide their advice repeatedly and ask whether the decision maker can learn to compare and rank the two forecasters based on past performance. We take an axiomatic approach and propose three natural axioms that a comparison test should comply with. We propose a test that complies with our axioms. Perhaps, not surprisingly, this test is closely related to the likelihood ratio of the two forecasts over the realized sequence of events. More surprisingly, this test is essentially unique. Furthermore, using results on the rate of convergence of supermartingales, we show that whenever the two experts\textquoteright{} advice are sufficiently distinct, the proposed test will detect the informed expert in any desired degree of precision in some fixed finite time.