Surrogate Scoring Rules

TL;DR

This paper introduces Surrogate Scoring Rules (SSR), a novel method for incentivizing truthful reporting of private probabilistic beliefs without access to ground truth, extending proper scoring rules to new settings.

Contribution

SSR extends proper scoring rules to settings lacking ground truth by using bias correction and error estimation, enabling truthful reporting and quality quantification.

Findings

SSR recovers SPSR in expectation with minimal prior information.

SSR induces dominant truthfulness in reporting.

Theoretical and empirical validation with human forecasters.

Abstract

Strictly proper scoring rules (SPSR) are incentive compatible for eliciting information about random variables from strategic agents when the principal can reward agents after the realization of the random variables. They also quantify the quality of elicited information, with more accurate predictions receiving higher scores in expectation. In this paper, we extend such scoring rules to settings where a principal elicits private probabilistic beliefs but only has access to agents' reports. We name our solution \emph{Surrogate Scoring Rules} (SSR). SSR build on a bias correction step and an error rate estimation procedure for a reference answer defined using agents' reports. We show that, with a single bit of information about the prior distribution of the random variables, SSR in a multi-task setting recover SPSR in expectation, as if having access to the ground truth. Therefore, a salient feature of SSR is that they quantify the quality of information despite the lack of ground truth, just as SPSR do for the setting \emph{with} ground truth. As a by-product, SSR induce \emph{dominant truthfulness} in reporting. Our method is verified both theoretically and empirically using data collected from real human forecasters.