Carrot and Stick: Eliciting Comparison Data and Beyond

TL;DR

This paper introduces peer prediction mechanisms with bonus-penalty payments to truthfully elicit comparison data from individuals, applicable to machine learning tasks like ranking and reinforcement learning from human feedback, supported by theoretical analysis and real-world experiments.

Contribution

It develops symmetrically strongly truthful peer prediction mechanisms leveraging stochastic transitivity, extending to networked data with Ising model signals, and provides theoretical conditions and experimental validation.

Findings

Mechanisms ensure truth-telling as a strict Bayesian Nash equilibrium.

Designed mechanisms require only one comparison evaluation per individual.

Experimental results support theoretical claims on real-world datasets.

Abstract

Comparison data elicited from people are fundamental to many machine learning tasks, including reinforcement learning from human feedback for large language models and estimating ranking models. They are typically subjective and not directly verifiable. How to truthfully elicit such comparison data from rational individuals? We design peer prediction mechanisms for eliciting comparison data using a bonus-penalty payment. Our design leverages on the strong stochastic transitivity for comparison data to create symmetrically strongly truthful mechanisms such that truth-telling 1) forms a strict Bayesian Nash equilibrium, and 2) yields the highest payment among all symmetric equilibria. Each individual only needs to evaluate one pair of items and report her comparison in our mechanism. We further extend the bonus-penalty payment concept to eliciting networked data, designing a symmetrically strongly truthful mechanism when agents' private signals are sampled according to the Ising models. We provide the necessary and sufficient conditions for our bonus-penalty payment to have truth-telling as a strict Bayesian Nash equilibrium. Experiments on two real-world datasets further support our theoretical discoveries.