Partial Truthfulness in Minimal Peer Prediction Mechanisms with Limited Knowledge

TL;DR

This paper investigates minimal peer prediction mechanisms with limited knowledge, characterizing the inefficiency of equilibrium strategies that are only partially truthful and providing mechanisms that achieve optimal bounds.

Contribution

It introduces a characterization of inefficiency in limited-knowledge peer prediction mechanisms and designs a mechanism that attains the theoretical bound.

Findings

Inefficiency of partial truthfulness scales as Θ(log n).

A peer prediction mechanism achieves this bound in expectation.

Equilibrium strategies are only partially truthful under limited knowledge.

Abstract

We study minimal single-task peer prediction mechanisms that have limited knowledge about agents' beliefs. Without knowing what agents' beliefs are or eliciting additional information, it is not possible to design a truthful mechanism in a Bayesian-Nash sense. We go beyond truthfulness and explore equilibrium strategy profiles that are only partially truthful. Using the results from the multi-armed bandit literature, we give a characterization of how inefficient these equilibria are comparing to truthful reporting. We measure the inefficiency of such strategies by counting the number of dishonest reports that any minimal knowledge-bounded mechanism must have. We show that the order of this number is $\Theta(\log n)$, where $n$ is the number of agents, and we provide a peer prediction mechanism that achieves this bound in expectation.