Partial Information Framework: Model-Based Aggregation of Estimates from Diverse Information Sources

TL;DR

This paper introduces a new model-based framework for aggregating estimates from diverse information sources in prediction polls, emphasizing information heterogeneity over measurement noise, and demonstrates its superior performance on real-world data.

Contribution

It formalizes information diversity as a key factor in estimate heterogeneity and proposes a practical model that improves aggregation accuracy in prediction polling.

Findings

Model outperforms standard measurement-error-based aggregators

Information diversity is a more significant source of heterogeneity

Framework effectively aggregates probability and point estimates

Abstract

Prediction polling is an increasingly popular form of crowdsourcing in which multiple participants estimate the probability or magnitude of some future event. These estimates are then aggregated into a single forecast. Historically, randomness in scientific estimation has been generally assumed to arise from unmeasured factors which are viewed as measurement noise. However, when combining subjective estimates, heterogeneity stemming from differences in the participants' information is often more important than measurement noise. This paper formalizes information diversity as an alternative source of such heterogeneity and introduces a novel modeling framework that is particularly well-suited for prediction polls. A practical specification of this framework is proposed and applied to the task of aggregating probability and point estimates from two real-world prediction polls. In both cases our model outperforms standard measurement-error-based aggregators, hence providing evidence in favor of information diversity being the more important source of heterogeneity.