Crowdsourcing with Unsure Option

TL;DR

This paper investigates the use of an unsure option in crowdsourcing to improve label quality and reduce costs, analyzing when it is effective and proposing an adaptive online mechanism for threshold setting.

Contribution

It provides an analysis of when unsure options lead to cost savings and introduces an online mechanism for confidence threshold selection in crowdsourcing.

Findings

Unsure option can reduce costs under certain conditions.

A new online mechanism for confidence threshold setting is proposed.

Guidelines for when to use unsure options effectively.

Abstract

One of the fundamental problems in crowdsourcing is the trade-off between the number of the workers needed for high-accuracy aggregation and the budget to pay. For saving budget, it is important to ensure high quality of the crowd-sourced labels, hence the total cost on label collection will be reduced. Since the self-confidence of the workers often has a close relationship with their abilities, a possible way for quality control is to request the workers to return the labels only when they feel confident, by means of providing unsure option to them. On the other hand, allowing workers to choose unsure option also leads to the potential danger of budget waste. In this work, we propose the analysis towards understanding when providing the unsure option indeed leads to significant cost reduction, as well as how the confidence threshold is set. We also propose an online mechanism, which is alternative for threshold selection when the estimation of the crowd ability distribution is difficult.