An Empirical Comparison of Algorithms for Aggregating Expert Predictions

TL;DR

This paper empirically compares various algorithms for aggregating expert predictions on NFL game outcomes, finding that Bayesian variance estimation outperforms simple averaging in accuracy and quadratic loss.

Contribution

It provides a comprehensive empirical evaluation of online and offline algorithms for expert prediction aggregation using real-world sports data.

Findings

Bayesian variance estimation performs best among tested algorithms.

Simple averaging is hard to beat in prediction accuracy.

Room for improvement exists in quadratic loss metrics.

Abstract

Predicting the outcomes of future events is a challenging problem for which a variety of solution methods have been explored and attempted. We present an empirical comparison of a variety of online and offline adaptive algorithms for aggregating experts' predictions of the outcomes of five years of US National Football League games (1319 games) using expert probability elicitations obtained from an Internet contest called ProbabilitySports. We find that it is difficult to improve over simple averaging of the predictions in terms of prediction accuracy, but that there is room for improvement in quadratic loss. Somewhat surprisingly, a Bayesian estimation algorithm which estimates the variance of each expert's prediction exhibits the most consistent superior performance over simple averaging among our collection of algorithms.