Surrogate Scoring Rules
Yang Liu, Juntao Wang, Yiling Chen

TL;DR
This paper introduces Surrogate Scoring Rules (SSR), a novel method for incentivizing truthful reporting of private probabilistic beliefs without access to ground truth, extending proper scoring rules to new settings.
Contribution
SSR extends proper scoring rules to settings lacking ground truth by using bias correction and error estimation, enabling truthful reporting and quality quantification.
Findings
SSR recovers SPSR in expectation with minimal prior information.
SSR induces dominant truthfulness in reporting.
Theoretical and empirical validation with human forecasters.
Abstract
Strictly proper scoring rules (SPSR) are incentive compatible for eliciting information about random variables from strategic agents when the principal can reward agents after the realization of the random variables. They also quantify the quality of elicited information, with more accurate predictions receiving higher scores in expectation. In this paper, we extend such scoring rules to settings where a principal elicits private probabilistic beliefs but only has access to agents' reports. We name our solution \emph{Surrogate Scoring Rules} (SSR). SSR build on a bias correction step and an error rate estimation procedure for a reference answer defined using agents' reports. We show that, with a single bit of information about the prior distribution of the random variables, SSR in a multi-task setting recover SPSR in expectation, as if having access to the ground truth. Therefore, a…
Peer Reviews
No public reviews on file for this paper yet. If you reviewed it on a platform where reviews are public (OpenReview, ICLR, NeurIPS, ICML), you can paste yours below so the community can read it here.
Videos
No videos yet. Explain this paper in a talk, walkthrough, or lecture? Add one.
Taxonomy
TopicsData Mining Algorithms and Applications
