A Collaborative Mechanism for Crowdsourcing Prediction Problems

TL;DR

This paper introduces a Crowdsourced Learning Mechanism that leverages collaborative hypothesis updating and betting incentives, aiming to improve prediction task crowdsourcing beyond traditional competitions.

Contribution

It proposes a novel collaborative framework inspired by prediction markets, addressing incentive issues in existing crowdsourcing prediction methods.

Findings

Participants profit based on their contribution to improving predictions.

The mechanism encourages collaborative learning through wagering.

Potential for more effective crowdsourced prediction than traditional competitions.

Abstract

Machine Learning competitions such as the Netflix Prize have proven reasonably successful as a method of "crowdsourcing" prediction tasks. But these competitions have a number of weaknesses, particularly in the incentive structure they create for the participants. We propose a new approach, called a Crowdsourced Learning Mechanism, in which participants collaboratively "learn" a hypothesis for a given prediction task. The approach draws heavily from the concept of a prediction market, where traders bet on the likelihood of a future event. In our framework, the mechanism continues to publish the current hypothesis, and participants can modify this hypothesis by wagering on an update. The critical incentive property is that a participant will profit an amount that scales according to how much her update improves performance on a released test set.