From Proper Scoring Rules to Max-Min Optimal Forecast Aggregation

TL;DR

This paper establishes a theoretical link between proper scoring rules and forecast aggregation methods, introducing quasi-arithmetic pooling as a unified framework with optimality and learnability properties.

Contribution

It introduces quasi-arithmetic pooling as a novel aggregation method derived from proper scoring rules, connecting incentive-compatible elicitation with forecast aggregation.

Findings

QA pooling with quadratic and logarithmic scores corresponds to linear and logarithmic aggregation.

Using QA pooling maximizes worst-case profit for agents subcontracting experts.

Aggregator scores are concave in expert weights, enabling low-regret online learning.

Abstract

This paper forges a strong connection between two seemingly unrelated forecasting problems: incentive-compatible forecast elicitation and forecast aggregation. Proper scoring rules are the well-known solution to the former problem. To each such rule $s$ we associate a corresponding method of aggregation, mapping expert forecasts and expert weights to a "consensus forecast," which we call *quasi-arithmetic (QA) pooling* with respect to $s$. We justify this correspondence in several ways: - QA pooling with respect to the two most well-studied scoring rules (quadratic and logarithmic) corresponds to the two most well-studied forecast aggregation methods (linear and logarithmic). - Given a scoring rule $s$ used for payment, a forecaster agent who sub-contracts several experts, paying them in proportion to their weights, is best off aggregating the experts' reports using QA pooling with respect to $s$, meaning this strategy maximizes its worst-case profit (over the possible outcomes). - The score of an aggregator who uses QA pooling is concave in the experts' weights. As a consequence, online gradient descent can be used to learn appropriate expert weights from repeated experiments with low regret. - The class of all QA pooling methods is characterized by a natural set of axioms (generalizing classical work by Kolmogorov on quasi-arithmetic means).