Building Robust Crowdsourcing Systems with Reputation-aware Decision Support Techniques

TL;DR

This paper introduces reputation-aware decision support techniques for crowdsourcing, addressing trust and capacity limitations to improve system robustness and social welfare.

Contribution

It proposes two novel trust-aware interaction decision-making approaches, SWORD and DRAFT, that account for workers' limited capacities in crowdsourcing systems.

Findings

Theoretical analysis shows social welfare can be optimized by tuning a key parameter.

SWORD and DRAFT improve task delegation efficiency in centralized and distributed systems.

Framework handles realistic scenarios with workers having varied capabilities.

Abstract

Crowdsourcing refers to the arrangement in which contributions are solicited from a large group of unrelated people. Due to this nature, crowdsourcers (or task requesters) often face uncertainty about the workers' capabilities which, in turn, affects the quality and timeliness of the results obtained. Trust is a mechanism used by people to facilitate interactions in human societies where risk and uncertain are common. The crucial challenge to building a robust crowdsourcing system is how to make trust-aware task delegation decisions to efficiently utilize the capacities of workers (or trustee agents) to achieve high social welfare? This book presents the research addressing this challenge. It goes beyond the existing trust management research framework by removing a widespread assumption implicitly adopted by existing research: that a trustee agent can process an unlimited number of interaction requests per discrete time unit without compromising its performance as perceived by the task requesters (or truster agents). Decision support in crowdsourcing is re-formalized as a multi-agent trust game based on the principles of the Congestion Game, which is solved by two trust-aware interaction decision-making approaches: 1) the Social Welfare Optimizing approach for Reputation-aware Decision-making (SWORD) approach, and 2) the Distributed Request Acceptance approach for Fair utilization of Trustee agents (DRAFT). SWORD is designed for centralized systems, while DRAFT is designed for fully distributed systems. Theoretical analyses have shown that the social welfare produced by these two approaches can be made closer to optimal by adjusting only one key parameter. With these two approaches, the framework of research for crowdsourcing systems can be enriched to handle more realistic scenarios where workers have varied and limited capabilities.