Reconciling Individual Probability Forecasts

TL;DR

This paper demonstrates that two parties with shared data cannot substantially disagree on individual probability forecasts because disagreements can be empirically tested and improved through an iterative reconciliation process, leading to near agreement.

Contribution

The paper introduces a method for reconciling individual probability models through empirical falsification and iterative improvement, resolving the model multiplicity problem.

Findings

Disagreements in individual probabilities can be empirically tested.

Iterative reconciliation leads to models that almost agree everywhere.

The process ensures models are both improved and mutually acceptable.

Abstract

Individual probabilities refer to the probabilities of outcomes that are realized only once: the probability that it will rain tomorrow, the probability that Alice will die within the next 12 months, the probability that Bob will be arrested for a violent crime in the next 18 months, etc. Individual probabilities are fundamentally unknowable. Nevertheless, we show that two parties who agree on the data -- or on how to sample from a data distribution -- cannot agree to disagree on how to model individual probabilities. This is because any two models of individual probabilities that substantially disagree can together be used to empirically falsify and improve at least one of the two models. This can be efficiently iterated in a process of "reconciliation" that results in models that both parties agree are superior to the models they started with, and which themselves (almost) agree on the forecasts of individual probabilities (almost) everywhere. We conclude that although individual probabilities are unknowable, they are contestable via a computationally and data efficient process that must lead to agreement. Thus we cannot find ourselves in a situation in which we have two equally accurate and unimprovable models that disagree substantially in their predictions -- providing an answer to what is sometimes called the predictive or model multiplicity problem.