Robust Forecast Aggregation

TL;DR

This paper develops methods for aggregating probabilistic forecasts from multiple Bayesian experts, proposing schemes with low regret in certain cases and analyzing limitations when many experts are involved.

Contribution

It introduces a framework for evaluating forecast aggregation schemes using scoring rules and regret, with new constructions for low regret schemes in specific expert settings.

Findings

Low regret aggregation schemes for two experts with Blackwell-ordered or conditionally i.i.d. signals

No aggregation scheme outperforms a 50-50 forecast when many experts with i.i.d. signals are involved

Aggregation performance depends critically on the number and information structure of experts

Abstract

Bayesian experts who are exposed to different evidence often make contradictory probabilistic forecasts. An aggregator, ignorant of the underlying model, uses this to calculate her own forecast. We use the notions of scoring rules and regret to propose a natural way to evaluate an aggregation scheme. We focus on a binary state space and construct low regret aggregation schemes whenever there are only two experts which are either Blackwell-ordered or receive conditionally i.i.d. signals. In contrast, if there are many experts with conditionally i.i.d. signals, then no scheme performs (asymptotically) better than a $(0.5,0.5)$ forecast.