Elicitation for Aggregation

TL;DR

This paper explores how to effectively elicit and combine probabilistic predictions from multiple Bayesian agents by capturing their confidence levels, enabling accurate aggregation with minimal sample access.

Contribution

It introduces a method to elicit confidence via hyperparameters from conjugate priors, facilitating optimal aggregation of agents' predictions with limited samples.

Findings

One sample suffices for aggregation if each posterior is uniquely identified by the hyperparameter.

The uniqueness of the hyperparameter enables successful aggregation using proper scoring rules.

A novel mechanism is proposed for cases where the hyperparameter uniqueness does not hold, with two samples.

Abstract

We study the problem of eliciting and aggregating probabilistic information from multiple agents. In order to successfully aggregate the predictions of agents, the principal needs to elicit some notion of confidence from agents, capturing how much experience or knowledge led to their predictions. To formalize this, we consider a principal who wishes to elicit predictions about a random variable from a group of Bayesian agents, each of whom have privately observed some independent samples of the random variable, and hopes to aggregate the predictions as if she had directly observed the samples of all agents. Leveraging techniques from Bayesian statistics, we represent confidence as the number of samples an agent has observed, which is quantified by a hyperparameter from a conjugate family of prior distributions. This then allows us to show that if the principal has access to a few samples, she can achieve her aggregation goal by eliciting predictions from agents using proper scoring rules. In particular, if she has access to one sample, she can successfully aggregate the agents' predictions if and only if every posterior predictive distribution corresponds to a unique value of the hyperparameter. Furthermore, this uniqueness holds for many common distributions of interest. When this uniqueness property does not hold, we construct a novel and intuitive mechanism where a principal with two samples can elicit and optimally aggregate the agents' predictions.