Eliciting Expertise without Verification

TL;DR

This paper addresses the challenge of eliciting truthful expertise from agents in crowdsourcing when answers cannot be directly verified, by designing mechanisms that incentivize effort and honesty based on agents' knowledge of others' beliefs.

Contribution

It introduces a natural model where more sophisticated agents understand less sophisticated agents' beliefs and develops novel mechanisms to promote effort and truthful reporting in this setting.

Findings

Mechanisms successfully incentivize effort and honesty.

The mechanisms output a correct hierarchy of information.

Effective in both single and multiple question scenarios.

Abstract

A central question of crowd-sourcing is how to elicit expertise from agents. This is even more difficult when answers cannot be directly verified. A key challenge is that sophisticated agents may strategically withhold effort or information when they believe their payoff will be based upon comparison with other agents whose reports will likely omit this information due to lack of effort or expertise. Our work defines a natural model for this setting based on the assumption that \emph{more sophisticated agents know the beliefs of less sophisticated agents}. We then provide a mechanism design framework for this setting. From this framework, we design several novel mechanisms, for both the single and multiple question settings, that (1) encourage agents to invest effort and provide their information honestly; (2) output a correct "hierarchy" of the information when agents are rational.