Stochastically Dominant Peer Prediction

TL;DR

This paper introduces a new concept called stochastically dominant truthfulness for peer prediction mechanisms, ensuring truthful reporting is robust across various utility functions, and proposes mechanisms that achieve this property with improved sensitivity.

Contribution

It defines SD-truthfulness as a stronger incentive guarantee, shows existing mechanisms lack this property, and proposes new mechanisms that enforce SD-truthfulness while maintaining high sensitivity.

Findings

The proposed enforcement via score rounding can ensure SD-truthfulness.

A new enforced agreement mechanism guarantees SD-truthfulness in binary-signal settings.

Empirically, the new mechanism achieves the highest sensitivity among SD-truthful mechanisms.

Abstract

Eliciting reliable human feedback is essential for many machine learning tasks, such as learning from noisy labels and aligning AI systems with human preferences. Peer prediction mechanisms incentivize truthful reporting without ground truth verification by scoring agents based on correlations with peers. Traditional mechanisms, which ensure that truth-telling maximizes the expected scores in equilibrium, can elicit honest information while assuming agents' utilities are linear functions of their scores. However, in practice, non-linear payment rules are usually preferred, or agents' utilities are inherently non-linear. We propose stochastically dominant truthfulness (SD-truthfulness) as a stronger guarantee: the score distribution of truth-telling stochastically dominates all other strategies, incentivizing truthful reporting for a wide range of monotone utility functions. Our first observation is that no existing peer prediction mechanism naturally satisfies this criterion without strong assumptions. A simple solution -- rounding scores into binary lotteries -- can enforce SD-truthfulness, but often degrades sensitivity, a key property related to fairness and statistical efficiency. We demonstrate how a more careful application of rounding can better preserve sensitivity. Furthermore, we introduce a new enforced agreement (EA) mechanism that is theoretically guaranteed to be SD-truthful in binary-signal settings under mild assumptions, and empirically achieves the highest sensitivity among all known SD-truthful mechanisms.