Design and Evaluation of Crowd-sourcing Platforms Based on Users Confidence Judgments

TL;DR

This paper investigates whether incorporating users' confidence judgments, based on their metacognitive abilities, can enhance the accuracy and performance of crowd-sourcing systems through mathematical and experimental analysis.

Contribution

It introduces a novel approach to improve crowd-sourcing accuracy by leveraging users' confidence levels and metacognition, supported by both theoretical and experimental validation.

Findings

Confidence-based answer weighting improves accuracy.

Metacognitive awareness correlates with answer correctness.

Using confidence judgments enhances crowd-sourcing performance.

Abstract

Crowd-sourcing deals with solving problems by assigning them to a large number of non-experts called crowd using their spare time. In these systems, the final answer to the question is determined by summing up the votes obtained from the community. The popularity of using these systems has increased by facilitation of access to community members through mobile phones and the Internet. One of the issues raised in crowd-sourcing is how to choose people and how to collect answers. Usually, the separation of users is done based on their performance in a pre-test. Designing the pre-test for performance calculation is challenging; The pre-test questions should be chosen in a way that they test the characteristics in people related to the main questions. One of the ways to increase the accuracy of crowd-sourcing systems is to pay attention to people's cognitive characteristics and decision-making model to form a crowd and improve the estimation of the accuracy of their answers to questions. People can estimate the correctness of their responses while making a decision. The accuracy of this estimate is determined by a quantity called metacognition ability. Metacoginition is referred to the case where the confidence level is considered along with the answer to increase the accuracy of the solution. In this paper, by both mathematical and experimental analysis, we would answer the following question: Is it possible to improve the performance of the crowd-sourcing system by knowing the metacognition of individuals and recording and using the users' confidence in their answers?