Probabilistic elicitation of expert knowledge through assessment of computer simulations

TL;DR

This paper introduces a novel probabilistic elicitation method that uses binary expert responses to assess simulated data, enabling flexible belief representation without requiring explicit opinion on parameters.

Contribution

It develops a new nonparametric framework for expert knowledge elicitation through simulation assessment, incorporating active learning and diagnostic tools.

Findings

Methods accurately captured expert beliefs about distributions.

Successfully distinguished differences in voter dispersion between countries.

Validated on both simulated and real expert judgments.

Abstract

We present a new method for probabilistic elicitation of expert knowledge using binary responses of human experts assessing simulated data from a statistical model, where the parameters are subject to uncertainty. The binary responses describe either the absolute realism of individual simulations or the relative realism of a pair of simulations in the two alternative versions of out approach. Each version provides a nonparametric representation of the expert belief distribution over the values of a model parameter, without demanding the assertion of any opinion on the parameter values themselves. Our framework also integrates the use of active learning to efficiently query the experts, with the possibility to additionally provide a useful misspecification diagnostic. We validate both methods on an automatic expert judging a binomial distribution, and on human experts judging the distribution of voters across political parties in the United States and Norway. Both methods provide flexible and meaningful representations of the human experts' beliefs, correctly identifying the higher dispersion of voters between parties in Norway.